Chip designer Cerebras Systems presents its Wafer Scale Engine 3: an AI processor the size of an entire wafer, good for 900,000 cores.

The chip designer Cerebras Systems thinks that everything could be a little bigger. The company is introducing its Wafer Scale Engine 3 (WSE-3): a giant chip that is the same size as an entire wafer. That provides space for four trillion transistors, good for 900,000 AI cores and 44GB of SRAM on the chip itself. Cerebras claims FP16 performance of 125 petaflops.

The chip measures 46,225 mm², compared to “only” 814 mm² for the also chunky Nvidia H100. Cerebras says its solution can train complex AI models about eight times faster. That’s not much since the chip is more than fifty times larger, but there are even more advantages. For example, a chip cluster can consume 1.2 PB of memory and process models with 24 trillion parameters, compared to 40 billion parameters for the DGX H100. The comparison stimulates the imagination, but is of course a bit of a comparison of apples and oranges.

Own supercomputer

Cerebras is building the chip into its own CS-3 supercomputer, which can handle clusters of up to 2,048 nodes. CS-3 would only need one day to train Llama 70B. Additionally, it would be easier to train such models on infrastructure. Cerebras isn’t finished yet: WSE-3 and CS-3 are, as the names suggest, the third iterations of the architecture.

Cerebras Systems is seeing interest from various industries in its giant chip and associated HPC solution. The adaptation appears interesting for training ever larger basic LLMs, which require ever more powerful and efficient systems.