Apple's Mac cluster hits 36 tokens a second, but long prompts wait 40 seconds

Apple ran Meta's 816 billion parameter Muse Spark Open across four M5 Ultra Mac Studios at over 36 tokens a second. Its tables also show a 40.6-second wait before the first word at 65,536 tokens. With two Macs, tensor splitting takes longer than pipeline splitting.

Apple says four Mac Studios can run a frontier model faster than most people read. In a research post dated 15 October, it reports "over 36 tokens per second" from Meta's Muse Spark Open, a model with 816 billion parameters. A token is a small chunk of text, roughly a word.

The headline measures only decoding. Decode is the speed at which words appear once the model has started. Prefill is the time spent reading the prompt before the first word appears. Apple's own tables put prefill at 40.6 seconds for a 65,536-token prompt on the best configuration. That wait determines whether the cluster is useful for agentic coding, where an AI tool edits files in many steps.

Apple's tables show how far apart the two measures sit. Four M5 Ultra machines split each layer of the model across the cluster, a method called tensor split. With an 8,192 token prompt, the first word arrives in 4.8 seconds and decode runs at 36.73 tokens a second.

Longer prompts barely change decode speed, but they stretch the wait. At 65,536 tokens, decode runs at 34.66 tokens a second. The first word takes 40.6 seconds. Apple says agentic coding applications often need contexts that large.

The two-machine results are less intuitive. Split by pipeline, each Mac holds a block of layers, and the pair reach the first word of that long prompt in 47.6 seconds. Split by tensor, the same pair take 70.3 seconds, which is 48% longer.

Tensor split is still the faster decoder, at 23.84 tokens a second against 14.37. Apple attributes the slower start to communication overhead. It writes that with few machines, the cost of tensor parallelism can offset some of its compute benefit at very large contexts.

Including both phases makes the cluster slower than its headline. Apple's benchmark generates 128 tokens after a 65,536-token prompt, and four Macs finish in about 44 seconds.

Two Macs take about 56 seconds split by pipeline and about 76 split by tensor. Both trail the four Mac result, as expected with half the machines.

The same-method comparison shows a large gain. Going from two Macs to four, both split by tensor, cuts the first-word wait from 70.3 to 40.6 seconds, which is 1.73 times faster. Decode rises from 23.84 to 34.66 tokens a second, which is 1.45 times.

Against the best two Mac setup, pipeline split, the four Mac cluster is only 1.17 times faster to the first word, but 2.4 times faster at decode.

The reason is structural, and Apple states it. Prefill is compute bound, so processor speed limits it. Decode is limited by memory bandwidth, the rate at which data moves out of memory. The M5 Ultra addresses the first limit with new neural accelerators, which Apple says cut time to first token by up to 4.6 times against an M3 Ultra.

It addresses the second with memory bandwidth of 1.2 terabytes a second, against 819 gigabytes on the M3 Ultra. That is a gain of about 1.5 times.

Apple measures decode up 1.6 times, in line with that gain. Compute improved about three times as much as bandwidth did, yet the long wait remains. A 65,536 token prompt is a great deal of reading, and on four Macs that works out at roughly 1,600 prompt tokens a second.

A 40-second wait hardly matters for a short chat, where the model then streams faster than anyone reads. Agentic tools edit files and resend changed context, so prefill is paid again whenever the start of the context changes. Apple itself frames the use case as agentic coding with large contexts.

The model sets a minimum cluster size. Apple says Muse Spark Open needs more than 1.6 terabytes in its native BF16 format, a 16-bit number format. Compressed to 4-bit NVFP4, it needs about 493 gigabytes, which still does not fit on one machine.

A single Mac Studio can hold a smaller dense model, one that uses all its parameters for every token. Apple lists Mistral Medium 3.5, at 128 billion parameters, and it decodes at 7.7 tokens a second.

In Apple's table, that model takes 24.6 seconds to start at 8,192 tokens. At 65,536 tokens the single machine waits 273 seconds. On four Macs that dense model needs 80 seconds, so it is slower to start than Muse Spark Open on the same cluster.

The wiring matters as well. Apple says macOS 26.2 introduced a low latency RDMA driver over Thunderbolt 5. RDMA is a direct memory to memory transfer method. The four machine mesh runs two cables between each pair.

Going from one cable to two cut the 65,536-token wait from about 50.5 seconds to 40.6. The interconnect therefore carries part of the load.

Apple adds that dense architectures scale better across several computers. Muse Spark Open has 42 billion of its 816 billion parameters active for each token, so it is not dense. Apple still finds that tensor split pays off as machines are added, but the bill rises in whole machines.

The Mac Studio M5 Ultra starts at $5,499, according to Macworld, with 96 gigabytes. Four of those hold only 384 gigabytes between them.

That is short of the 493 gigabyte weights. Apple's specification page lists 256 and 512 gigabyte configurations, so a working cluster is four 256 gigabyte machines or two 512 gigabyte ones, each priced above the base.

A four Mac cluster keeps the model and its prompts on local hardware and adds no fee per token. The price is about 44 seconds for a large context turn that returns 128 tokens.

A single Mac Studio on a 128 billion parameter dense model is slower still on long prompts. A hosted GPU endpoint is quicker but sends the code away. So the cluster fits chat and short context, and for agentic coding it is a machine that works in seconds to minutes a step.

Meta's own post pitches two 512 gigabyte M5 Ultra Mac Studios as the desk setup. On Apple's numbers, that pair reaches the first word in 47.6 seconds with pipeline split, against 40.6 on four.

At that 65,536-token size, the pair decodes at 14.4 tokens a second against 34.7 on four. The two-machine pitch is therefore the version to judge by, while the 36 tokens a second headline belongs to the four-machine cluster at an 8,192-token prompt.