NVIDIA's Groq 3 LPX Achieves 3,400 Tokens/Second, 4x Faster Than Competitors in Agentic AI Push

What Nvidia's first Groq 3 LPU benchmarks do and don't tell us about its $20B gamble

Nvidia's $20 billion bet on Groq's LPU tech sure looks like it was a good one. On Monday, the GPU giant offered the first glimpse of just how big a speedup its Groq 3-based LPX racks will provide. In an independent benchmark conducted by Artificial Analysis, Nvidia's LPX rack systems managed to churn out 3,400 tokens a second (tok/s) with a 100,000-token input sequence in Google's Gemma 4 31B model. According to Nvidia, this makes it 4x faster than the nearest alternative platform, which going off Artificial Analysis' leaderboard would be a direct dig at Cerebras, which managed a still impressive 882 tok/s under the same conditions. Acquihired by Nvidia in late December, Groq has LPUs that feature an SRAM-heavy dataflow architecture designed specifically for high-performance inference serving. Unlike traditional datacenter GPUs, which rely on high-speed DRAM memory tech like GDDR7 and HBM4, Groq's chips rely entirely on a large pool of on-die SRAM orders of magnitude faster than even the best HBM stacks (around 2.75 TB/s each) available today. And when it comes to inference, memory bandwidth is the bottleneck to beat. The third generation of the chips launched as part of Nvidia's broader Vera Rubin platform earlier this year boasted 150 TB/s of memory bandwidth. The catch, of course, is that SRAM chews up a ton of die area, which means you can't pack all that much into a reticle-sized part. Where Nvidia's top-specced Rubin GPU boasts 288 GB of onboard memory, each Groq 3 LPU has just 500 MB -- 576x less. That's far too little memory to run Gemma 4 31B on a single LPU, so Nvidia's architecture uses Ethernet to distribute models across multiple accelerators. Each LPX rack can be equipped with up to 256 LPUs for 128 GB of high bandwidth SRAM. For large models, multiple LPX racks can be ganged together. So why would anyone need to run Gemma 4 31B at 3,400 tokens a second? You probably don't, but the AI code assistant or agent you're using it with might. The faster you can generate tokens, the longer models can reason, the more turns the agent can take, and the more information it can process or actions it can take in the same window of time. The big bet here is that, in the agentic age, faster inference servers make for smarter agents and are therefore worth a premium. And the combination of Nvidia GPUs and Groq 3 LPUs certainly seems to have caught Nvidia's customers' attention. On Monday, the GPU giant revealed that Netherlands-based neocloud Nebius would be among the first to field the combined systems in its datacenters. Just how impressive is Nvidia's LPX system really? While 3,400 tok/s is impressive at face value, Gemma 4 31B is arguably a best case scenario for the hardware, and it remains to be seen how gracefully the architecture will scale to larger, more complex MoE models. At 31 billion parameters, the model fits neatly into a single LPX rack regardless of what data type Nvidia is using to store the weights. Nvidia is running the model at FP8, which means it needs a little over 31 GB or just under 64 LPUs of SRAM capacity. We've reached out to Nvidia for comment on how exactly it's distributing the model across those chips, but didn't get a clear answer. Our best guess is plain old pipeline parallelism with possibly a dash of data parallelism to improve performance at higher concurrency levels. Pipeline parallelism, if you're not familiar, distributes the model across the accelerators and then processes decode operations in one great big pipeline. Since there are enough LPUs, by our approximation, to hold four copies of the model, data parallelism would effectively duplicate this pipeline and provide additional independent slots for higher concurrency. If you're thinking that 31B isn't very large, you're right. The model is small enough to run on a high-end consumer graphics card like an RTX 3090/4090 -- at 4-bit precision of course. While it may be a small model, we wouldn't write off Nvidia's test results just yet. The model itself is relatively small, but it's a dense model, which means all 31 billion parameters are activated each time a token is generated. Thirty-one billion also happens to be fairly close to the number of active parameters used by much larger MoE models like DeepSeek's 671 billion parameter V3, which has 37 billion active parameters. It should be noted, however, that because MoE models use different parameters for each token generated, they suffer from performance overheads that you don't see in dense models. The other problem worth noting is that to serve a model like DeepSeek V3 on an LPX system would require 1,342 accelerators or just over 5 LPX racks. That's a lot of accelerators to wrangle. Taking LPX to the next level The performance figures shared by Nvidia today are for a model running entirely on LPUs. However, Nvidia's biggest performance gains will come from combining its GPUs with the dataflow accelerators in a heterogeneous inference architecture. We discussed this approach in more detail earlier this year, but, in a nutshell, what Nvidia is doing is breaking the inference workload into two distinct phases. The compute heavy prefill phase, during which prompts are processed and the key-value caches used to track model state are generated, is done on its GPUs, while the memory-bandwidth intensive decode phase is offloaded to the Groq 3 LPUs. The result is a massive speedup in inference performance beyond what GPUs alone could deliver. The chart below summarizes how Nvidia's GPUs and LPUs fit together nicely. Rubin's high compute density makes it ideal for high volume, bulk inference where the goal is reasonable interactivity (tok/s per user) for a large number of concurrent users, while LPUs are best suited for maximum interactivity. By combining the two, Nvidia has managed to lift the Pareto curve up and to the right, maximizing throughput without compromising on interactivity. The 3,400 tok/s recorded by Artificial Analysis reflects performance on the far right of the chart, but by combining Groq's LPUs with its GPUs, Nvidia expects it can achieve a decent fraction of that performance across a large number of concurrent users. Nvidia's figures may not age well Along with the LPUs' potential scaling challenges, it's also worth discussing Nvidia's comparison against Cerebras' chips. Nvidia claims its systems are 4x faster than Cerebras' parts, which is true. Artificial Analysis testing at a 100,000 input sequence (prompt) does corroborate this. What is left out of that conversation is the number of chips required. We don't know the specifics of Cerebras' config, but it's not hard to suss out. Cerebras usually serves its models at mixed precision (eight and 16 bits) with 16-bit activations, which means the entire Gemma 4 31B model should fit into one, at most two 44 GB CS-3 accelerators, whereas Nvidia needs at least 64 chips. Cerebras' performance figures also don't reflect its next-generation accelerators announced last week. The WSE-3T-based CS-4 accelerator doubles the compute, I/O bandwidth, fabric speed, and memory bandwidth of its existing chips, and also triples the number of accelerators per rack. The new systems are expected to launch on Cerebras' inference cloud later this year alongside a heterogeneous GPU+WSE-based system similar to Nvidia's NVL72+LPX config we discussed earlier. The tie up, announced at AMD's Advancing AI event last month, will see the House of Zen combine its Helios GPU rack systems with Cerebras dinner-plate accelerators. Cerebras is working with AWS on a similar config as well. As such, Nvidia's Gemma 4 31B performance comparison may not age well. ®

Nvidia says Groq racks will be online this year following $20 billion purchase

* Nvidia announced that its Groq 3 LPX chip is in full production, marking the commercialization of technology from the company's largest-ever acquisition. * Nvidia's race to manufacture the chip and make it available to customers highlights the importance of low-latency inference that's needed to make AI agents feel responsive. * In December, the chipmaker bought assets from Groq for $20 billion. In this article * NVDA Follow your favorite stocksCREATE FREE ACCOUNT The Nvidia Groq 3 LPU chip during the Nvidia GTC conference in San Jose, California, March 18, 2026. David Paul Morris | Bloomberg | Getty Images Nvidia announced on Monday that its Groq 3 LPX rack is in full production, marking the commercialization of technology from the company's largest acquisition on record. The Groq rack will be deployed alongside Vera central processors and Rubin graphics processors at neocloud Nebius, and will be online later this year, Nvidia senior director Dion Harris told reporters. Nvidia's race to manufacture Groq's chip and make it available to customers highlights the growing importance of low-latency inference that's needed to make AI agents feel responsive without long lags for users, especially for coding. Cloud companies can charge more for these kind of tokens, Nvidia says. "For folks who are serving tokens, it unlocks the ability to offer premium tiers of service for those users and those customers who actually demand the most latency-sensitive" service agreements, Harris said on the call. In December, Nvidia bought assets from chip startup Groq for $20 billion, the company's largest purchase.. The Groq architecture includes 500 megabytes of speedy SRAM on the chip's die itself to reduce memory-related bottlenecks. Groq chips are manufactured by Samsung, while Taiwan Semiconductor Manufacturing makes Nvidia's GPUs. Nvidia packages 256 individual Groq 3 chips into its LPX racks. Nvidia said that its Groq 3 LPX rack can deliver 3,400 tokens per second, citing a benchmark from Artificial Analysis. It's a competitive space. Smaller GPU maker Advanced Micro Devices announced earlier this year it would integrate its rack-scale systems with chips from Cerebras, which recently went public, focusing on low-latency inference. OpenAI's newly announced Ultrafast mode currently promises 750 tokens per second, and is "powered by Cerebras." Low-latency chips don't replace the GPU, the workhorse of AI chips, which can do training as well as inference and are flexible enough to adapt to new technologies and models. Low-latency chips like Groq mainly focus on a part of serving models called the "decode" phase. "This isn't about replacing GPUs," Harris said. "It's about using the right price, right processor for the right part of the workload," Nvidia is currently ramping up shipments of its Vera Rubin systems, which started production earlier this year. At the Vera Rubin and Groq 3 LPX unveiling in March, Nvidia CEO Jensen Huang projected $1 trillion in cumulative sales between the current-generation Blackwell chips and the new Vera Rubin systems, through 2027. Huang said at the time he would allocate a quarter of data center space intended for coding applications to Groq chips. "The rest of my data center is all 100% Vera Rubin," Huang said. Nvidia reports earnings on Wednesday. watch now VIDEO13:5913:59 First look at Vera Rubin, Nvidia's next AI system that's 10 times more efficient Tech Choose CNBC as your preferred source on Google and never miss a moment from the most trusted name in business news.

With Groq 3 LPX in Full Production, NVIDIA Extends Vera Rubin Inference for Agents

The next era of AI inference won't be defined by a single breakthrough chip, network or system. It'll be defined by how every layer of the AI factory works together. That's why NVIDIA is extending Vera Rubin NVL72 with fast token generation for agentic systems. Announced today, the NVIDIA Vera Rubin rack-scale system NVIDIA Groq 3 LPX is in full production. In an Artificial Analysis benchmark running Gemma 4 31B, an open source agentic model, it delivered 3,400 output tokens per second for 100,000-token long-context use cases critical to agentic systems, 4x faster than the nearest alternative platform. Industry partners worldwide are adopting Vera Rubin platform solutions. SpaceXAI announced that NVIDIA Vera CPUs will power its next generation of agentic AI. CoreWeave has deployed into production Spectrum-X Multiplane, which connects NVIDIA Vera Rubin racks using multiple parallel switches to provide high-bandwidth, flat and lossless AI networks. Nebius is the first AI cloud to adopt NVIDIA Groq 3 LPX. As AI shifts from training to reasoning and agentic, inference has become the new frontier. Agentic AI systems are generating more tokens, processing dramatically larger context windows and increasingly collaborating with other AI systems to solve complex problems. These workloads demand a new class of infrastructure optimized not just for performance but for throughput, responsiveness and economics at unprecedented scale. At the Hot Chips conference this week in Palo Alto, California, NVIDIA is showcasing how extreme codesign is reshaping the AI factory from end to end. By architecting compute, networking and inference acceleration as a unified system, NVIDIA is helping customers build infrastructure purpose-built for the emerging demands of long-context inference and multi-agent systems. Extreme Codesign Optimizes for Performance Extreme codesign is the guiding principle behind NVIDIA platforms. Vera Rubin is engineered to accelerate inference as agents reason over increasingly long sequences. NVIDIA Spectrum-X Ethernet moves those massive data flows efficiently across AI factories, and NVIDIA Groq 3 LPX is built to generate tokens at ultrafast speeds. Together, they show how NVIDIA is optimizing every stage of the AI pipeline, from context and communication to generation, as part of a single, integrated AI factory architecture. NVIDIA Groq 3 LPX brings a new low-latency inference architecture designed to work alongside Vera Rubin NLV72, the most versatile AI factory platform, helping enterprises and cloud providers deliver the low latency, extreme throughput and scalable economics required for agentic applications. Breakthrough performance comes not from optimizing individual components in isolation, but from codesigning every layer of the stack. From networking and context processing to large-scale inference, NVIDIA's full-stack platform turns AI factories into integrated engines for intelligence, built to turn ever-growing volumes of tokens into revenue. Tuesday, Aug. 24, 8:00 a.m. PT 🔗 NVIDIA Partners Adopt Vera Rubin for Lowest Token Costs Nebius, a leading AI cloud, is first to adopt NVIDIA Groq 3 LPX, giving developers access to leading token generation speeds for highly responsive agentic AI applications. Adding NVIDIA Groq 3 LPX to NVIDIA Vera Rubin NVL72 in Nebius Token Factory will boost inference performance so developers can build highly interactive agents, coding systems and other real-time AI experiences at scale. Connecting NVIDIA Vera Rubin racks, CoreWeave is deploying Spectrum-X Multiplane in production, unlocking advances for its AI cloud infrastructure. Tuesday, Aug. 24, 8:00 a.m. PT 🔗 SpaceXAI Adopts NVIDIA Vera CPUs for Agentic AI SpaceXAI plans to build and scale its future AI architecture around NVIDIA Vera Rubin, from data centers on Earth to orbital satellites. The company plans to deploy NVIDIA Vera CPUs to accelerate the CPU-intensive work behind agentic AI, including orchestration, tool use, code execution, data processing and simulation. The SpaceXAI partnership extends NVIDIA's full-stack AI platform to SpaceXAI, bringing together Vera CPUs, NVIDIA accelerated computing, networking and software to advance AI at unprecedented scale. Designed for the agentic era, Vera Rubin provides leading per-core performance, exceptional memory bandwidth and predictable performance under load, helping agents complete tasks faster and keeping valuable GPU infrastructure fully utilized. Tuesday, Aug. 24, 8:00 a.m. PT 🔗 NVIDIA Groq 3 LPX: The Interactive AI Inference Accelerator Codesigned with the Vera Rubin NVL72 platform, NVIDIA Groq 3 LPX is helping AI factories deliver tokens at the lowest latency for agentic workloads. Agentic AI is creating a new performance challenge: decode latency. As AI agents reason, use tools and interact with other systems, they generate responses one token at a time, causing even tiny delays to multiply across complex chains of work. To keep agents operating at the pace users expect, NVIDIA Groq 3 LPX extends the Vera Rubin NVL72 platform with specialized acceleration for token generation. NVIDIA Rubin GPUs handle large-scale context processing while LPX accelerates latency-sensitive decode workloads. The result is faster, more predictable token generation that helps AI factories deliver responsive reasoning, smoother agent interactions and greater infrastructure efficiency. Together, Rubin GPUs and LPUs are designed to eliminate the traditional tradeoff between speed and throughput, helping AI providers deliver responsive, large-scale inference for the next generation of agentic AI applications. Building the Token Factory As the industry shifts from model training to serving intelligence at scale, infrastructure must evolve into what NVIDIA describes as a "token factory" capable of delivering performance, throughput, intelligence integrity and economic efficiency simultaneously. Agentic AI systems increasingly communicate with other AI systems, access multiple data sources and maintain large amounts of context, creating unprecedented demand for fast inference. NVIDIA Groq 3 LPX was designed for exactly these workloads. As an extension of the Vera Rubin NVL72, it enables ultrafast responsiveness even across massive context windows while helping service providers maximize throughput and infrastructure utilization. Extreme Codesign for Inference Unlike standalone accelerators, NVIDIA Groq 3 LPX combines the strengths of GPUs and LPUs through extreme codesign. Rubin GPUs and LPUs jointly compute every layer of an AI model, enabling new levels of inference performance for agentic workloads. At scale, fleets of LPUs operate as a giant processor optimized for deterministic inference. A rack-scale NVIDIA Groq 3 LPX deployment can include 256 LP30 accelerators connected through direct chip-to-chip links, creating a highly efficient inference engine built for modern AI factories. Designed for the Agentic AI Era As reasoning models grow and agentic workflows generate ever more tokens, the infrastructure required to serve them must evolve. NVIDIA Groq 3 LPX extends the Vera Rubin NVL72 platform with a purpose-built inference architecture designed to maximize responsiveness, throughput and efficiency, helping power the next generation of AI factories. And this is only the beginning, more optimizations, more models, more performance when paired with Vera Rubin NVL72 -- new levels of throughput and interactivity are coming. Stay tuned. Tuesday, Aug. 24, 8:00 a.m. PT 🔗 NVIDIA Spectrum-X Multiplane Enables Massive AI Factory Scale on a Flatter, More Resilient Network As AI factories grow massive, the network has become a critical engine of performance. At Hot Chips, NVIDIA is spotlighting Spectrum-X Multiplane -- the latest in the hardware-accelerated Spectrum-X Ethernet architecture that lets Ethernet scale to unprecedented size while avoiding the latency, jitter and cost of adding another network tier. NVIDIA Spectrum-X Ethernet is designed as an end-to-end, AI-optimized Ethernet platform, combining NVIDIA Spectrum-X Ethernet switches, SuperNICs and software to improve the performance and efficiency of Ethernet-based AI infrastructure for AI factories and clouds. The platform is designed to deliver 1.6x better AI networking performance compared with off-the-shelf Ethernet, while providing consistent, predictable performance in multi-tenant environments. Multiplane Unlocks Scale Without the Tradeoffs of a New Tier Scaling an AI factory beyond today's largest clusters traditionally means adding a third network tier, which adds latency, slows things down unpredictably and drives up the cost of cabling, optics and power. Spectrum-X Multiplane takes a simpler approach: It splits each server's network connection into several independent paths, or "planes," each running its own lightweight two-tier network. The result is a flat, simple network that scales to 512,000 GPUs, without the added cost and complexity of a third tier. This all happens automatically. A dedicated hardware engine inside the NVIDIA ConnectX SuperNIC manages traffic across the planes and instantly reroutes around any failure, so applications and software simply see one fast, reliable connection. In an eight-plane topology, if one plane fails, the network still maintains about 90% of its total bandwidth, with hardware recovery that's 11x faster than software-based multiplane load balancing. This translates to 1.6x higher AI factory output. Built Through Extreme Codesign That reliability comes from extreme codesign of Vera Rubin NVL72, spanning switch silicon, SuperNICs and software. Spectrum-X SN6000 series switches, based on the 102.4Tb/s Spectrum-6 Ethernet ASIC and ConnectX-9 SuperNICs, supporting up to 1,600Gb/s per GPU, are purpose-built for Vera Rubin NVL72 AI factories. Spectrum-XGS Ethernet extends that same codesign across data centers, letting multiple facilities function as a single AI super-factory and accelerating multi-site NCCL collectives by 1.9x. Tuesday, Aug. 24, 8:00 a.m. PT 🔗 NVIDIA Introduces Scale-In Infrastructure for Agentic AI Factories, Powered by BlueField-4, DOCA NVIDIA is introducing NVIDIA Scale-In, the fifth pillar of NVIDIA AI networking and a new class of accelerated network infrastructure for agentic AI factories. Scale-In extends purpose-built acceleration to the infrastructure services that secure, manage and operate the AI factory. Powered by the NVIDIA BlueField-4 processor and NVIDIA DOCA software platform and connected over NVIDIA Spectrum-X Ethernet, NVIDIA Scale-In transforms the traditional north-south access network into a unified, accelerated infrastructure domain. Cloud computing brought software-defined networking, composability and elasticity to the data center, enabling users, applications, data and services to scale dynamically. Agentic AI represents the next platform shift. AI factories bring together massive accelerated compute with growing numbers of users, applications and autonomous agents, all continuously interacting with data, storage and services. This transforms the demands on the infrastructure that brings AI to life. Networking, storage, cybersecurity and operations must now be accelerated alongside AI compute, combining software-defined flexibility with purpose-built hardware acceleration and full-stack codesign. NVIDIA Scale-In delivers multi-tenant networking, high-performance storage access, in-silicon security, elastic provisioning and real-time observability, while keeping infrastructure processing independent of host compute resources. By accelerating and codesigning these services as part of the AI factory, Scale-In helps security, data access and operations scale alongside AI compute. The result is secure, efficient and manageable shared infrastructure for deploying and operating agentic AI at massive scale. Tuesday, Aug. 24, 8:00 a.m. PT 🔗 NVIDIA NVLink Fusion Connects XPUs to NVIDIA's Leading AI Platform NVIDIA NVLink Fusion brings custom silicon into NVIDIA's world-leading AI infrastructure platform, enabling hyperscalers and AI-native companies to build semi-custom AI factories with greater performance, flexibility and speed. As AI models grow in size and complexity, raw compute alone is not enough. AI factories require high-bandwidth, low-latency scale-up networking, proven rack-scale architectures and a full ecosystem spanning power, cooling, management software and supply chain. NVLink Fusion addresses these challenges by connecting custom XPUs and CPUs to NVIDIA's scale-up and scale-out technology stack. The platform includes sixth-generation NVIDIA NVLink and NVLink Switch purpose-built scale-up networking, as well as NVLink-C2C for energy-efficient connectivity between XPUs and CPUs. Through the NVIDIA MGX ecosystem, adopters can also use production-proven rack designs, components, manufacturing partner solutions and open, extensible software for distributed computing, disaggregated workloads and cluster management. By standardizing GPU- and XPU-based systems on a unified architecture, NVLink Fusion helps decouple data center buildout from silicon readiness. Operators can share rack footprints, networking, cooling, power delivery and management systems, then adjust the mix of GPUs and XPUs as supply and workload requirements evolve. NVLink Fusion extends the NVIDIA AI platform's vertically integrated, horizontally open approach to custom silicon. It gives partners the freedom to innovate where they differentiate while drawing on NVIDIA technologies across compute, networking, infrastructure and software -- creating a single, flexible AI factory that no one company could build alone.

Up to 30x More Work Per Watt: NVIDIA Vera Rubin NVL72 Sets a New Efficiency Standard for AI Agents

New on-silicon performance data measured by NVIDIA using real-world agentic coding trajectories shows Vera Rubin NVL72 systems deliver 30x higher throughput per megawatt and 35x lower token costs than NVIDIA GB300 NVL72. According to OpenRouter data, agentic AI workloads consume 15x more tokens than a simple chat request. Why? Consider what happens when an AI agent researches a company for an investment decision. The agent queries financial databases, searches news and filings, invokes a sub-agent to run peer comparisons and model valuations, then synthesizes everything into a recommendation. Agents and sub-agents keep reasoning until the task is done, driving increased token demand. With every step, the accumulated tokens become the input to the next, making long-context handling central to agentic AI performance. The same pattern plays out across every agentic use case, from software development to customer service to deep research. As agentic AI moves into production across industries, the infrastructure running it needs to meet that token demand efficiently. New measured performance data shows NVIDIA Vera Rubin NVL72 systems deliver up to 30x higher throughput per megawatt than NVIDIA GB300 NVL72 on agentic workloads. NVIDIA measured this inference throughput data using the SemiAnalysis AgentX workload, consisting of recorded real-world agentic coding sessions, with actual context growth, tool calls and sub-agent spawning preserved. For power-constrained AI factories, that translates directly into 30x more agentic work for the same energy footprint. These early results for Vera Rubin NVL72 demonstrate NVIDIA's accelerated pace of innovation. With continuous software optimizations, performance across both Vera Rubin NVL72 and GB300 NVL72 will continue to improve. Vera Rubin NVL72: 30x Higher Throughput per Megawatt and 35x Lower Token Cost Agentic workloads look fundamentally different from chat or document summarization, where input and output sequences typically range from 1K to 8K tokens. In agentic sessions, context accumulates across steps and can reach hundreds of thousands of input tokens, with wide variability in both input and output lengths across requests. Performance measurement must evolve to capture the full agent workflow rather than a single inference request. The results below reflect performance measured on real-world agentic coding trajectories. In SemiAnalysis AgentX, the NVIDIA Blackwell platform delivers leading performance across multiple agentic models including Kimi K3, MiniMax M3, GLM5.3, Qwen3.5 and DeepSeek V4 Pro. For example, GB300 NVL72 delivers up to 15x better throughput per megawatt than the NVIDIA Hopper architecture on the DeepSeek V4 Pro model, giving customers a high-performance foundation to run agentic workloads. This leap reflects the advantage of a larger scale-up GPU domain and codesigned software in delivering significantly better inference efficiency. Vera Rubin extends that advantage, lifting the performance across the entire Pareto curve, to deliver as much as 30x higher throughput per megawatt than GB300 NVL72 on the DeepSeek V4 Pro model. These early results, measured using the SemiAnalysis AgentX workload and currently pending SemiAnalysis review, don't yet reflect Vera CPU performance for tool calling. NVIDIA DSX MaxLPS technologies manages power across the GPU, rack and workload levels to provision up to 40% more GPUs within the same megawatt budget, pushing throughput per megawatt further at AI factory scale. Throughput per megawatt also directly impacts the cost of every token produced. At up to 35x lower cost per million tokens than GB300 NVL72, Vera Rubin NVL72 can run agents continuously, at scale, across the full breadth of customers' workloads. For power-constrained AI factories, throughput per megawatt determines AI factory revenue and cost per million tokens determines the profit margin on that revenue. Extreme Codesign for Agentic Scale Modern inference optimization spans a range of techniques that are especially critical for agentic AI. NVIDIA Vera Rubin NVL72 enables all of these and more through extreme codesign across every layer of the platform to deliver multifold performance gains. * Disaggregated serving separates context processing (prefill) from response generation (decode) so each scales independently. * Rate matching synchronizes the speeds at which prefill GPUs and decode GPUs produce tokens to maximize efficiency. * Large-scale expert parallelism distributes expert sub-networks in mixture-of-experts models across the scale-up GPU domain. * Distributed KV-caching extends memory across the scale-up GPU domain, while KV-cache offloading tiers less-active context to host and storage, keeping previously processed context accessible without recomputation. * KV-aware routing directs incoming requests to the GPUs that already hold the relevant cached context, reducing redundant computation across long sessions. * Fused CUDA kernels like MegaMoE combine many computation and inter-GPU communication operations into a single execution pass, keeping GPUs active rather than waiting for data. NVIDIA Rubin GPUs' enhanced fifth-generation Tensor Cores and the third-generation Transformer Engine accelerate both prefill and decode stages of inference. NVFP4 quantization compresses model weights to 4-bit precision, reducing memory footprint and increasing throughput without sacrificing output quality. The NVL72 scale-up domain, a defining architecture across Vera Rubin and Grace Blackwell, enables the high-bandwidth and low-latency inter-GPU communication essential for techniques such as large-scale expert parallelism and distributed KV-caching. Purpose-built to power this scale-up domain, NVIDIA NVLink interconnect technology and NVLink Switches, now in their sixth generation, deliver 10x higher packet rates and 3x lower latency than off-the-shelf Ethernet alternatives. Spanning optimized CUDA kernels, inference runtimes like NVIDIA TensorRT LLM and serving frameworks like NVIDIA Dynamo, NVIDIA's software stack is codesigned with the hardware to enable inference optimizations. While the results above reflect current Vera Rubin NVL72 performance, the full platform is a seven-chip architecture that also includes the NVIDIA Vera CPU, Groq 3 LPU, NVLink 6 Switch, BlueField-4 DPU, Spectrum-6 SPX and ConnectX-9 SuperNIC, all purpose-built for AI factories deploying agents at scale. Extreme codesign also extends to NVIDIA's co-engineering with its partners. Vera Rubin is in full production and is scaling across the ecosystem. Learn more about the NVIDIA Vera Rubin platform.

NVIDIA Groq 3 LPX Now in Full Production With World-Class Speed for Agentic AI

Groq 3 LPX Extends the Vera Rubin Platform, Delivering Ultrafast Token Generation for the Next Generation of Agentic AI, With Nebius the First to Adopt * In Artificial Analysis benchmarking, Groq 3 LPX showcased world-class speed for agentic coding and other latency-sensitive workloads. * NVIDIA Groq 3 LPX extends the inference performance of NVIDIA Vera Rubin NVL72 systems by dramatically increasing token generation rates. * Nebius is the first AI cloud to adopt NVIDIA Groq 3 LPX. Hot Chips -- NVIDIA today announced that NVIDIA Groq 3 LPX, the interactive AI inference accelerator, is now in full production. An extension of the NVIDIA Vera Rubin platform, Groq 3 LPX delivers a major boost in AI inference by enabling ultrafast token generation for highly responsive agentic systems. Agentic systems can generate massive volumes of tokens across hundreds or thousands of inference steps, making faster token generation critical for agents to reason, act and complete complex tasks in real time. Vera Rubin NVL72 systems provide the most versatile training and inference platform for every AI factory. NVIDIA Groq 3 LPX extends the inference performance of Vera Rubin NVL72 by dramatically increasing the rate of token generation, providing premium user experiences for context-heavy workloads so agents can act at extreme speeds. NVIDIA Groq 3 LPX is pushing the frontier of AI inference. It delivered a record 3,400 output tokens per second in Artificial Analysis benchmarking running Gemma 4 31B, an open source agentic model, with a 100,000-token context critical for agentic systems -- the fastest performance ever recorded for the model. Groq 3 LPX enables agentic tasks such as coding in minutes versus hours, providing 4x faster responsiveness for agents and latency-sensitive workloads than the nearest alternative platform. "Inference is the growth engine of AI. NVIDIA Grace Blackwell and NVL72 revolutionized large language model inference with an unprecedented leap in performance and efficiency," said Jensen Huang, founder and CEO of NVIDIA. "Vera Rubin extends that vision with workload-optimized AI factory configurations designed for the era of agentic AI, advancing the performance frontier with LPX for ultrafast token generation. This transforms how intelligence is produced, delivering another giant leap in AI throughput, efficiency and responsiveness, just as demand for AI computation is accelerating worldwide." Groq 3 LPX -- The Interactive AI Inference Accelerator Agentic AI creates two distinct computing challenges: efficiently processing enormous amounts of context and generating tokens with extremely low latency. NVIDIA Groq 3 LPX is purpose-built to extend Vera Rubin's interactivity -- the rate at which tokens are generated for an individual user, determining how quickly an agent can complete each step of its work. Faster generation gives agents more time to inspect files, write and test code, call tools, verify results and iterate while maintaining a responsive user experience. AI Cloud Momentum for Groq 3 LPX AI clouds are becoming the engines of the AI economy, giving enterprises and developers access to advanced infrastructure for training, reasoning and inference at scale. For providers serving latency-sensitive, high-volume inference workloads, NVIDIA Groq 3 LPX provides a path to deploy differentiated compute in proven rack-scale systems. Nebius, a leading AI cloud, plans to bring NVIDIA Groq 3 LPX to Nebius Token Factory, its production inference platform, giving developers access to extreme token generation speed for highly responsive agentic AI applications. "Generation is the phase of inference that determines how responsive an AI system actually is, and that's exactly what NVIDIA Groq 3 LPX is built to accelerate," said Danila Shtan, chief technology officer of Nebius. "As the first AI cloud bringing it to production via Nebius Token Factory, we're making sure every step of an agent's loop feels instant -- through the same API developers are already using, with no migration to a new stack." Following Nebius, purpose-built AI inference cloud Groq plans to be among the platform's earliest adopters. Extreme Codesign for AI Factories Through extreme codesign across seven chips and five purpose-built racks, NVIDIA Vera Rubin is the most extensive AI factory platform. NVIDIA Vera Rubin NVL72 and Groq 3 LPX tackle the various workload requirements of customer AI factories, including frontier model makers and open model service providers. These rack platforms feature NVIDIA BlueField®-4 DPUs and work in combination with NVIDIA Vera CPU racks, NVIDIA Vera BlueField-4 STX storage and NVIDIA Spectrum™-6 SPX Ethernet to optimize multi-agent systems for the highest throughput per watt and the lowest-latency inference.

Nvidia's dedicated inference accelerator Groq 3 LPX enters full production to supercharge AI agents

Nvidia's dedicated inference accelerator Groq 3 LPX enters full production to supercharge AI agents Chipmaker Nvidia Corp. says its dedicated artificial intelligence inference accelerator Groq 3 LPX has now entered full production as it strives to maintain its dominance in the world of AI compute. The new chip, announced today at Hot Chips 2026, is described as a purpose-built extension to Nvidia's flagship Vera Rubin data center platform. According to Nvidia, it's designed to deliver ultra-fast token generation speeds, which are necessary to run highly responsive agentic AI workloads. The chipmaker said the neocloud provider Nebius Group N.V. has already signed on as the first customer to commit to using the new chip. Inference is the AI industry's lingo for the process of running fully trained AI models in production, and it's increasingly focused on autonomous AI agents that can perform tasks on behalf of humans. These agents must be able to do everything from reason and plan, write and execute code, inspect system files and use third-party tools in continuous loops. They can quickly crunch through thousands of tokens across these complex chains, but this sometimes results in massive "decode latency" that can create frustrating delays. To prevent this from happening, AI data centers need more specialized compute architectures that can disaggregate the enormous context processing from token generation, in order to increase the speed at which AI agents can reason and work. This is where the Vera Rubin NVL72 rack-scale platform comes in. It's powered by dozens of Nvidia's most powerful Vera Rubin graphics processing units, which are designed to handle the large-scale context ingestion and processing. Now, with the addition of Groq 3 LPX, it can offload those decode workloads onto the new accelerators. According to Nvidia, Groq 3 LPX makes it possible for a full rack-scale deployment to harness up to 256 LP30 accelerators, linked by its ultra-high-bandwidth chip interconnects. It means the LPUs can work in tandem with the GPUs to compute every step in an AI agent's chain of reasoning, acting as a unified inference engine designed for enterprise scale. Nvidia said this will help to eliminate the tradeoff between throughput and response times. Groq 3 LPX gives AI agents the ability to scan long context windows, verify data, call third-party tools and iterate on the most complex, multistep tasks in real-time, without creating delays for users. It has the third-party data to back up this claim, too. In benchmark tests performed by Artificial Analysis, Groq 3 LPX was shown to output a record-breaking 3,400 tokens per second when running the open-source Gemma 4 31B agentic model with a 100,000-token context window. The chipmaker reckons this makes it four times more responsive for latency-sensitive workloads compared to rival platforms, which means multistep agentic tasks can get done in minutes instead of hours. The new chip was built using technology licensed from a smaller chipmaker called Groq Inc. Nvidia paid the startup a stunning $20 billion in December to be allowed to access its tech, and also hired its founder Jonathan Ross and President Sunny Madra as part of that deal. Groq, not to be confused with SpaceX Corp.'s Grok AI model, develops processors specifically focused on inference rather than AI training. Nvidia founder and Chief Executive Jensen Huang said his company had already revolutionized AI inference performance and efficiency with its Grace Blackwell and NVL72 platforms. "Vera Rubin extends this with workload-optimized AI factory configurations designed for the era of agentic AI," he said. "We're advancing the performance frontier with LPX for ultra-fast token generation. This transforms how intelligence is produced, delivering another giant leap in AI throughput, efficiency and responsiveness." Nebius is one of the first companies to agree to deploy the Groq 3 LPX chips. It said it will use them in the Nebius Token Factory, its production inference platform, to provide customers with more extreme token generation speeds for their most responsive agentic applications. "Generation is the phase of inference that determines how responsive an AI system actually is, and that's exactly what Groq 3 LPX is built to accelerate," said Nebius Chief Technology Officer Danila Shtan. "As the first AI cloud to bring it to production via Nebius Token Factory, we're making sure every step of an agent's loop feels instant." Nvidia had plenty more to say at Hot Chips. It revealed it has signed up SpaceX as its latest flagship customer. The space rocket and AI company plans to build its next-generation AI architecture around the Vera Rubin platform. Specifically, it said it will deploy Nvidia's Vera central processing units for CPU-intensive orchestration, tool execution and simulation tasks that span terrestrial data centers and orbital satellites. The chipmaker also showcased several complementary technologies for AI factories, including Spectrum-X Multiplane, which is a new AI-optimized Ethernet architecture that divides server connections into parallel paths to scale enormous clusters of up to 512,000 GPUs. Then there's Nvidia Scale-In, which is a new infrastructure software platform that offloads and accelerates security, networking and data management from the main host compute nodes, and Nvidia NVLink Fusion, which enables custom CPUs and data processing units to connect to its sixth-generation NVLink rack systems.

NVIDIA Vera Rubin Records A Massive 30x Increase In Throughput Per Watt Than Blackwell While Offering a 35x Token Cost Reduction Across Agentic AI Workloads

NVIDIA's Vera Rubin platform offers disruptive token throughput at a much lower cost than Blackwell, showcasing its Agentic AI prowess. NVIDIA Software Stack Optimizations Continue To Propel Blackwell, But Vera Rubin Sits In A Whole Different League For Agentic AI Use Cases The latest on-silicon agentic AI performance results were measured by NVIDIA using real-world agentic coding trajectories. The benchmark used was SemiAnalysis AgentX, which evaluates AI infrastructure such as Vera Rubin in agentic-coding inference workloads across various models such as Kimi K3, MiniMax M3, GLM5.3, Qwen3.5, and DeepSeek V4 Pro. The following table explains the key metrics of AgentX: First up are the NVIDIA Blackwell results. A GB300 NVL72 "Grace Blackwell" server offers 15x higher throughput per MW than H200 NVL8 "Hopper" solutions in DeepSeek-v4-PRO 1.6T. The Blackwell solution sustains higher and more responsive agentic inference throughput within the same power budget. At the same time, NVIDIA Blackwell also offers 10x lower cost (per million tokens) than its predecessor. This lower TCO means that AI operators can support more agent capacity within the same power and infrastructure budget or deliver the same capacity with an even lower operating cost. Even more impressive for Blackwell is that it performs incredibly well as models scale. Using Kimi K3 2.8T, the Blackwell solution offered an 80x increase in throughput per MW versus Hopper, and sustained 215 tokens per second per user in terms of interactivity, which was well beyond what the Hopper solution is capable of. With Blackwell done, now it's time to move to look at the next chapter of Agentic AI and Inference: Vera Rubin. The NVIDIA Vera Rubin NVL72 platform delivers a massive 30x increase in Throughput versus Grace Blackwell NVL72 in DeepSeek-v4-PRO 1.6T. At around 160 tokens per second per user of interactivity, Vera Rubin simply blazes past Grace Blackwell and also achieves up to ~280 TPS per user, whereas Blackwell peaks out below 180 TPS per user. Vera Rubin also offers significantly lower token cost in Agentic Coding workloads versus Blackwell. A 35x lower cost per million tokens shows that NVL72 can run several agents continuously at scale across a broad set of workloads. And with technologies such as NVIDIA DSX MaxLPS that manage power across GPUs, Racks, and Workloads Levels, AI factories can provision up to 40% more GPUs within the same Megawatt budget. Furthermore, these benchmarks don't yet reflect Vera CPU performance for tool calling, and just focus on the Vera Rubin chips rather than the complete "Extreme Codesign" 7-chip platform. So as more Vera Rubin platforms come online, we will see even more impressive results around AI agents and inference. With that said, NVIDIA has now commenced production across all of its AI-based solutions, including Vera CPUs, Rubin GPUs, Vera Rubin servers, Groq 3 LPX chips, and the full range of networking technologies. Follow Wccftech on Google to get more of our news coverage in your feeds.

NVIDIA Enters Full Production of Groq 3 LPX AI Inference Accelerator Chips, Supercharging Vera Rubin With The Fastest Token Generation Speeds Ever Recorded

NVIDIA's Groq 3 LPX is now in full production, offering big token generation speedups for Vera Rubin platforms in the Agentic AI space. NVIDIA Dials Up Vera Rubin NVL72 Token Generation Capabilities, Recording 3,400 TPS With Groq 3 LPX AI Inference Accelerators As part of its Hot Chips 2026 announcements, NVIDIA today announced that its Groq 3 LPX AI inference accelerator chip is in full production. This announcement follows the mass production announcements of Vera CPUs and Vera Rubin servers, marking the robust execution of NVIDIA's AI roadmap. The Groq 3 LPX racks serve as an extension to the NVIDIA Vera Rubin platform, delivering boosted AI inference capabilities that enable ultra-fast token generation for response-sensitive agentic workloads, as these systems can generate a massive amount of tokens across hundreds or even thousands of inference steps, making fast token generation a critical step for AI factories. While NVIDIA's Vera Rubin NVL72 is the bleeding-edge solution for Agentic AI, its capabilities can be further bolstered by the Groq 3 LPX solution. In one demonstration, NVIDIA showcases its Vera Rubin NVL72 platform with Groq 3 LPX pushing out a record 3,400 tokens per second in Artificial Analysis running the Gemma 4 31B open model. With a context window of 100,000 tokens, this was the fastest performance ever recorded on the model. NVIDIA Rubin GPUs handle large-scale context processing while LPX accelerates latency-sensitive decode workloads. The result is faster, more predictable token generation that helps AI factories deliver responsive reasoning, smoother agent interactions, and greater infrastructure efficiency. With Groq 3 LPX, NVIDIA also enables agentic AI tasks such as coding to be done within minutes versus hours, offering a 4x boost in response times versus the nearest alternative platform. Big words right there! "Inference is the growth engine of AI. NVIDIA Grace Blackwell and NVL72 revolutionized large language model inference with an unprecedented leap in performance and efficiency," said Jensen Huang, founder and CEO of NVIDIA. "Vera Rubin extends that vision with workload-optimized AI factory configurations designed for the era of agentic AI, advancing the performance frontier with LPX for ultrafast token generation. This transforms how intelligence is produced, delivering another giant leap in AI throughput, efficiency and responsiveness, just as demand for AI computation is accelerating worldwide." The NVIDIA Groq 3 LPX solution is also being eyed by AI cloud providers as a solution to the growing needs, giving enterprises and developers access to advanced infrastructure for training, reasoning, and inference at scale. As such, Nebius plans to utilize Groq 3 LPX in its Nebius Token Factory. "Generation is the phase of inference that determines how responsive an AI system actually is, and that's exactly what NVIDIA Groq 3 LPX is built to accelerate," said Danila Shtan, chief technology officer of Nebius. "As the first AI cloud bringing it to production via Nebius Token Factory, we're making sure every step of an agent's loop feels instant -- through the same API developers are already using, with no migration to a new stack." Groq 3 LPX and Vera Rubin, as a combo, are designed for the Agentic AI era, bringing a purpose-built inference architecture designed to maximize responsiveness, throughput, and efficiency, helping power the next generation of AI factories. Follow Wccftech on Google to get more of our news coverage in your feeds.