Meta's Prometheus matches Llama 3's interruption rate, with spare GPUs as the cost

Meta's Prometheus reports more than 85% goodput on pretraining and more than 90% on reinforcement learning. Its interruptions per GPU match the 16,000 GPU Llama 3 cluster of 2024, while spare hardware carries the cost.

Meta says its Prometheus supercluster trained the Muse Wonder models at more than 85% goodput on pretraining. For reinforcement learning, the figure was more than 90%. Goodput is the share of time the GPUs spend on useful training.

Prometheus is a gigawatt scale site in New Albany, Ohio. It holds hundreds of thousands of GPUs today and is meant to reach roughly 1 million. At that size, every point of lost training time is a block of idle silicon.

The interruption rate is the sharper number. In the Meta Engineering post, the mean time between interruptions during the Muse Wonder runs was about 47,000 GPU hours. Firmware fixes since then have lifted it above 54,000.

On a base of 8,000 GPUs, that is 4.1 interruptions a day, then 3.6. Meta's own north star is 2 a day on that base.

Prometheus has recovered to the Llama 3 record rather than surpassed it. The Llama 3 paper logged 466 interruptions over 54 days of pretraining on 16,384 GPUs. That is one every 45,600 GPU hours, or 4.2 a day on the same 8,000 GPU base. Prometheus is within 3% of that on the first Muse Wonder figure and about 15% better on the later one. It reaches that rate with far more GPUs arranged in 72 GPU racks.

Its traffic also crosses buildings over links of 2 km to more than 100 km. Matching the Llama 3 rate at that scale is significant.

A rate per GPU that holds flat while the GPU count multiplies means more failures across the fleet. The defences are spare hardware, fault tolerant software and restarts from saved progress. The bill arrives as idle silicon or repeated work.

In a GB200 or GB300 contract, the useful price is per usable GPU hour. It should be backed by an interruption rate commitment, not set per nominal GPU. A rack sold as 72 GPUs and planned as 64 carries eight GPUs of insurance.

Skipping that insurance means paying for restarts after each fault. The buyer carries that cost unless the contract prices it in.

The spare hardware is large. For a training run of 32,768 active GPUs, Meta provisions 523 NVL72 racks, or 37,656 GPUs. Meta calls that about 13% above the active footprint.

The sum gives 14.9% above. The 13% is the share of the provisioned fleet that sits spare.

Most of the spare capacity is slice geometry. Each rack runs 64 GPUs and keeps eight in reserve. Meta holds roughly 2% more racks for whole rack faults.

Scale makes small percentages expensive. On a fleet of 100,000 GPUs, one point of goodput is worth 1,000 GPUs, or about 8.8 million GPU hours a year.

The 72 GPU design explains why spares matter. An NVL72 domain widens the fault blast radius ninefold over the eight GPU domains before it. One bad link or switch can now take down 72 GPUs rather than eight.

The causes have shifted with each platform. Meta says NVLink signal integrity was initially the primary bottleneck for GB200. The racks hold more than 600,000 NVLink connections, and even marginal degradation triggers recovery cycles.

Memory faults in HBM and LPDDR then became a major source of interruptions as Meta scaled GB300. Meta also lists NVLink switch health and backend network events among its categories. It gives no ranking of the causes today.

The Llama 3 record looked different. Faulty GPUs led at 30.1% of unexpected interruptions, and HBM3 memory followed at 17.2%.

Meta's 99% reduction is measured from a much worse start. In the first phase of Prometheus in mid 2025, interruptions peaked at nearly 1,000 events per 8,000 GPUs a day. By CTOL's calculation, that is a gap of roughly 192 GPU hours between failures.

The fix was operational as much as technical. Meta cites a triage team colocated at New Albany and a stability sprint in March that synchronised fixes across the stack. Every repaired system now sits in mandatory quarantine before it returns to the compute pool.

Firmware automation can push fixes to more than 100,000 GPUs within a few hours. Meta says the 90th percentile upgrade time for a single pod fell from more than seven hours to two. That speed lets it ship fixes for the top interruption categories quickly.

Software also limits the damage of a single fault. Parallelism aware fault tolerance lets one faulty data parallel replica go offline while the rest of the job continues. A fault that spans more than one replica still restarts the whole job from its last checkpoint.

Hardware screening starts before a rack takes traffic. Meta validates more than 600,000 NVLink paths with diagnostic stress tests before routing production work. Each NVL72 domain carries 5,184 signalling links at 200 Gbps, and every one is a possible point of failure.

At 54,000 hours, the fleet is about halfway to Meta's reliability target: two interruptions per 8,000 GPUs a day, or about 100,000 GPU hours between failures.

The next test is close. Meta says a third platform, based on Vera Rubin NVL72, is arriving soon at Prometheus.

The worst months of GB200 came at the start of its deployment. Meta itself says early volume exposes quality issues that stay hidden at lower counts. A new platform restarts that clock.

Meta reports third quarter results after the market close on Wednesday 21 October. That is the next scheduled chance for investors to ask how much of the Prometheus racks serve as spares rather than training.