Groq's Next-Gen LPU Adopts 3D Stacked DRAM to Support Expanding AI Models

Groq LPU Transitions to 3D Stacked DRAM Architecture

Groq is preparing its next-generation Language Processing Unit with a fundamental architectural shift from SRAM to 3D stacked DRAM technology. This change marks a strategic pivot toward memory capacity over bandwidth speed as AI models continue expanding in size. A semiconductor industry insider confirmed the move, stating that Groq plans to introduce 3D packaging to its next-generation chips, involving stacking DRAM rather than existing SRAM1

1

. The decision reflects the industry's growing need to accommodate larger AI models that demand more memory capacity for high-performance inference.

Current Groq LPU Specifications and Performance

The current generation Groq LPU packs 500 MB of SRAM, achieving bandwidth rates of 150 TB/s per chip. Each individual Groq LPU delivers 1.20 PFLOPs FP8 compute performance1

1

. When configured in LPX racks with 256 chips, the system delivers approximately 128 GB of SRAM capacity and 640 TB/s of scale-up bandwidth2

2

. These LPX racks are part of NVIDIA's Vera Rubin ecosystem and entered full production in August, reportedly delivering around 3,400 tokens per second when paired with NVL72 servers2

2

. The racks now operate alongside GroqCloud and Groq's role as an NVIDIA Cloud Partner.

Memory Capacity Over Bandwidth Trade-Off

The shift from SRAM to 3D stacked DRAM represents a calculated trade-off between speed and capacity. SRAM delivers blistering-fast bandwidth but offers limited capacity, whereas DRAM provides relatively slower speeds but enables implementation of much larger capacities1

1

. For anyone watching AI hardware scaling closely, this trade-off is direct: less of the extreme bandwidth Groq's current design is known for, but significantly more memory capacity from DRAM2

2

. As growing model sizes push hardware requirements higher, Groq's main focus appears to be memory capacity while retaining some aspects of the high bandwidth speeds from its SRAM-only chips1

1

.

3D Stacking Technology and HBM Similarity

With 3D stacked DRAM technology, Groq will stack multiple DRAM layers in a single vertical chip, connected using TSVs (Through-Silicon-Vias). This approach mirrors how HBM DRAM is designed today1

1

. The most likely implementation involves TSV-connected, HBM-style stacked DRAM, which should raise density and improve power efficiency2

2

. The 3D stacking enables reduced surface area and increased power efficiency compared to traditional DRAM packages1

1

. There are hints that Groq may keep its DRAM and SRAM chips separate from one another, offering the industry solutions tailored to specific needs1

1

.

NVIDIA Partnership and Market Position

Clarifying a common misconception, NVIDIA did not acquire Groq. In late 2025, the companies signed a non-exclusive licensing deal reportedly worth about $20 billion, with Groq remaining independent2

2

. NVIDIA hired several key Groq personnel, including founder Jonathan Ross, while Groq continued operating independently. The latest Groq chips might see increased adoption from NVIDIA, which has been using them to boost AI factories and AI platforms1

1

. Watch for next developments through Groq's data center and cloud platforms as model sizes keep climbing, particularly in inference workloads where the balance between capacity and speed becomes critical for serving large-scale AI models efficiently.