Meta Prometheus Supercluster Keeps 11% of Nvidia GPUs Idle to Prevent AI Training Failures

Meta Prometheus Supercluster Scales to 1 Million Nvidia GPUs

Meta's Prometheus supercluster in New Albany, Ohio represents one of the company's largest AI training infrastructure deployments, with plans to scale to approximately 1 million Nvidia GPUs consuming a gigawatt of power capacity1

1

2

2

. The facility currently runs hundreds of thousands of GPUs across multiple buildings, connected by miles of fiber optic cables. Two years ago, this site operated as a conventional data center handling web traffic before its transformation into a massive AI training facility. The supercluster trained Meta's Muse AI models, including the recently released Muse Wonder, and demonstrates the company's commitment to building infrastructure capable of supporting next-generation artificial intelligence development.

Over 11% of GPUs Reserved as Hot Spares to Maintain Training Continuity

Meta disclosed that more than 11% of the Nvidia GPUs at Prometheus are either dead or sitting idle as hot spares, waiting to step in when active GPUs fail1

1

. The facility uses Grace Blackwell GB200 and GB300 rack systems, with each NVLink domain containing 72 GPUs. However, only 64 GPUs remain active at any given moment, while eight serve as hot spares. When a training run requires 32,768 GPUs, Meta actually needs 37,656 GPUs deployed—approximately 13% more hardware than actively computing2

2

. This redundancy strategy prevents a single GPU failure from taking down an entire rack and interrupting the training run. Meta also keeps about 2% of complete racks in reserve. The massive pool of idle hardware represents the cost of maintaining continuity in large-scale AI training operations.

Interruptions in Training Runs Drop 99% Through Engineering Improvements

When Meta began the first phase of Prometheus deployment, the cluster experienced roughly 1,000 interruptions per day for every 8,000 GPUs—a staggering failure rate that threatened to derail AI training progress1

1

. Through a year of intensive engineering work, Meta cut interruptions by 99%, reducing them to approximately 11 per day per 8,000 GPUs2

2

. The company achieved this by implementing data parallelism, where multiple copies of the model exist within the cluster. When a fault affects only one data-parallel replica, that compromised slice goes offline and restarts on spare hosts while the rest of the cluster continues training. Meta also deployed comprehensive checkpointing to save model state, gradients, and optimizer data asynchronously to multi-tier storage. During Muse Wonder's training, the cluster maintained more than 85% goodput for pretraining and over 90% for reinforcement learning. The system now runs about 47,000 GPU hours on average between interruptions, recently improved to above 54,000 after firmware fixes, with a goal of reaching 100,000.

NVLink and Power Management Present Operational Challenges

The most difficult operational challenges emerged from managing over 600,000 NVLink connections that link 72 GPUs to function as one massive accelerator1

1

. Earlier H100 and H200-based systems featured just eight GPUs per NVLink domain with shorter, less numerous connections. Blackwell generation systems expanded this nine-fold, and even marginal signal degradation triggered frequent NVLink recovery cycles. Meta implemented continuous telemetry systems and conducted diagnostic burns to isolate weak links before putting systems into production. Signal quality on NVLink connections was the biggest early problem, while memory faults became more prominent as Meta scaled up GB300 racks2

2

. Power management also posed significant challenges. When tens of thousands of GPUs compute and communicate simultaneously, the cluster's power draw swings by 30% to 40%. Starting or pausing a large job can shift tens of megawatts in seconds. Meta developed software called Dynamo Dimmer to gradually cap GPU power, supported by batteries at rack, facility, and grid levels.

Three Generations of Nvidia Racks Power AI Training Infrastructure

Prometheus currently runs two Nvidia platforms, with a third on the way. The Catalina platform uses GB200 NVL72 racks, while Clemente uses the newer GB300 NVL72 systems2

2

. A third platform called San Miguel will add Nvidia's Vera Rubin NVL72 racks to the deployment. The GPUs operate across traditional data centers, AI-optimized buildings, and rapid-deployment structures. Each building's network connects up to 20,736 GPUs, while a separate aggregation network joins buildings over distances ranging from 2km to more than 100km. Meta also leases capacity from partners to supplement its owned infrastructure. The company is applying lessons learned from Prometheus to its next supercluster, Hyperion, being built in Richland Parish, Louisiana. Meta has pledged $1 billion to data center communities and claims to pay the full cost of energy used in New Albany, Ohio, matching it with new clean and renewable energy sources. However, in July, the company left the RE100 pact as its gas build-out for AI training outpaced its renewable energy procurement.