Meta Keeps 11% of Nvidia GPUs Idle in Prometheus Supercluster to Prevent AI Training Failures

Meta disclosed operational details about its Prometheus supercluster that expose the harsh realities of running AI training infrastructure at unprecedented scale. More than 11% of Nvidia GPUs at the New Albany, Ohio facility are either failed or kept idle as hot spares, waiting to replace the next component that drops out1

1

2

2

. This massive redundancy strategy represents one of several measures Meta has implemented to keep large-scale AI training runs on track as it scales toward roughly 1 million distributed GPUs3

3

.

From 1,000 Daily Interruptions to Near-Continuous Operation

When Meta began the first phase of its Prometheus deployment, the company experienced roughly 1,000 interruptions per day for every 8,000 GPUs1

1

. These interruptions forced AI model training runs to stop and roll back to the last known good checkpoint. A year of focused engineering work cut that figure by 99%, bringing interruptions down to about 11 per day per 8,000 GPUs2

2

. The cluster now runs about 47,000 GPU hours on average between interruptions, recently pushing above 54,000 after firmware fixes, with Meta targeting approximately 100,0003

3

. This improvement enabled Meta to train models like Muse Spark and Muse Wonder with minimal disruption.

Hot Spares and Redundancy Drive Continuity Costs

The Meta Prometheus supercluster currently runs a combination of GB200 and GB300 systems from Nvidia's Grace Blackwell generation, each featuring 72 GPUs per NVLink domain1

1

. However, only 64 GPUs are actively used in training at any moment, with the remaining eight serving as hot spares2

2

. Meta also holds about 2% of racks in reserve3

3

. This means a training run on 32,768 Nvidia GPUs actually requires 37,656 GPUs, representing about 13% overhead3

3

. When a GPU fails, a spare can immediately slot in, preventing an entire rack from dropping out of the cluster and minimizing interruption time.

Data Parallelism Prevents Total System Halts

Meta leveraged data parallelism in large-scale AI training to further reduce operational challenges. Multiple copies of the AI model training run simultaneously across different portions of the cluster. When a fault can be isolated to a single data-parallel model replica, Meta takes only that faulty replica offline while the remaining cluster continues productive training1

1

. The compromised slice restarts on available spare hosts and resynchronizes, converting what would have been a total system halt into a minor, temporary reduction in aggregate throughput. Training runs only get interrupted when faults exceed the data parallel domain, at which point Meta's comprehensive checkpointing scheme saves the day by preserving model state, gradients, and optimizer data to multi-tier storage2

2

.

NVLink Fabric Challenges at Blackwell Scale

Making the NVLink fabric connecting 72 GPUs behave like one unified accelerator presented significant infrastructure improvements challenges. Earlier H100 and H200-based systems featured just eight GPUs per NVLink domain with shorter, less numerous links. Nvidia's Grace Blackwell generation expanded this functionality nine-fold1

1

. With over 600,000 NVLink connections, even marginal signal degradation triggers frequent recovery cycles2

2

. Signal quality on these connections became the biggest early problem during mid-20253

3

. Meta implemented continuous telemetry systems and conducted L11 diagnostic burns to isolate weak links before putting systems into production, addressing these operational challenges head-on.

Power Transients and Energy Management at Gigawatt Scale

Ramping up and down tens of thousands of Nvidia GPUs for AI training infrastructure creates massive power transients that can overwhelm supply equipment1

1

. When tens of thousands of GPUs compute and communicate in lockstep, the cluster's power draw swings by 30% to 40%3

3

. Starting or pausing a large job can shift tens of megawatts in seconds. Meta developed software called Dynamo Dimmer to gradually cap GPU power, backed by batteries at rack, facility, and grid levels to smooth these fluctuations. The company says it pays the full cost of energy used in New Albany, Ohio and matches it with new clean and renewable energy, though it left the RE100 pact in July as its gas build-out for AI outpaced renewables3

3

.

Achieving High Goodput Despite Massive Scale

By focusing relentlessly on stability and reliability, Meta achieved more than 85% goodput for pre-training and more than 90% for reinforcement learning during Muse Wonder's development2

2

3

3

. Goodput measures useful throughput, distinguishing productive computation from wasted cycles due to failures and restarts. The Meta Prometheus supercluster spans multiple buildings in New Albany, Ohio, including traditional data centers, AI-optimized buildings, and rapid-deployment structures, all linked by miles of fiber3

3

. Each building's network connects up to 20,736 GPUs, with a separate aggregation network joining buildings over distances from 2km to more than 100km.

Three Generations and Future Expansion

Prometheus currently runs two Nvidia platforms: Catalina using GB200 NVL72 racks and Clemente using the newer GB300 NVL72 systems3

3

. A third platform called San Miguel will add Nvidia's Vera Rubin systems NVL72 racks to the mix. At full scale, the AI training infrastructure will comprise roughly 1 million distributed GPUs functioning in lockstep, consuming a gigawatt of capacity1

1

3

3

. Meta is already applying lessons learned to its next supercluster, Hyperion, being built in Richland Parish, Louisiana3

3

. The company has pledged $1 billion to data center communities as it continues expanding AI model training capacity. Watch for how Meta's approach to redundancy and checkpointing influences industry standards for large-scale AI training as competitors race to build similar infrastructure.