To train Llama 4, Meta uses a cluster of over 100,000 Nvidia H100 GPUs; probably a record.

Meta CEO Mark Zuckerberg says the Llama 4 Large Language Model is trained on a cluster with more than 100,000 Nvidia H100 GPUs. This cluster is immediately the largest in the world, which is at least being talked about publicly. Training LLMs requires a lot of computing power, and the largest models are built over months on AI supercomputers. These AI clusters are equipped with powerful AI accelerators tailored to the training load. The Nvidia H100 is the most powerful chip of its kind available at scale.

Big, bigger, biggest

Elon Musk is also said to be building (or having built) a cluster for xAI with a similar number of Nvidia H100 chips. The goal is to continue to expand this cluster in the long term, but currently the record appears to be around 100,000 chips.

It is difficult to estimate how much a data center with so many accelerators consumes annually. Calculate the equivalent of a city. Llama 4 will be one of the most powerful AI models of its kind and should be able to think rationally. Only time will tell whether these capabilities are worth the increasing impact on global power supplies. It is unclear how Meta’s cluster is powered.

Nuclear energy

Several data specialists, including Oracle, AWS and Microsoft, are turning to nuclear energy to satisfy AI’s energy hunger. The tech giants are not only considering purchasing existing reactors, but are also investing in their own small modular reactors to power the data centers of tomorrow.