Diffusion LLMs are not a good fit for distributed inference

When I first heard about diffusion LLMs, I wanted to see how well they would work in a distributed setup. In the years since creating Distributed Llama, which runs autoregressive LLMs such as Qwen3 across multiple machines, I have found that one of its biggest bottlenecks is the number of times the nodes must synchronize for each token. This initially made diffusion LLMs seem promising because they process multiple tokens per forward pass. However, I found that other properties of these models unfortunately make them a worse fit for distributed inference.

A few years ago, I would have started with some calculations to see whether the idea made sense. But with AI changing how we write code, I… just implemented it. Distributed Llama already had many of the features I needed, so I forked it instead of starting from scratch. After about eight hours of uninterrupted work, Codex produced the first version that could correctly run Diffusion Gemma inference on 1, 2, 4, and 8 nodes. It took a few more days to polish the code. Then it was time to run the benchmarks.

I tested the fork on a cluster of four Raspberry Pi 5 devices, each with 8 GB of RAM.

TP-Link LS1008G Switch
      | | | |
      | | | |_______ Raspberry Pi 5 (ROOT)
      | | |_________ Raspberry Pi 5 (WORKER 1)
      | |___________ Raspberry Pi 5 (WORKER 2)
      |_____________ Raspberry Pi 5 (WORKER 3)

And the results were… disappointing.

	Tokens per second	Data transferred / token	Data transferred per canvas / forward pass
2 nodes	0.353	7.6 MB	139.2 MB
4 nodes	0.709	22.9 MB	419.7 MB

Diffusion Gemma has 4B active parameters. For comparison, Qwen3-30B-A3B has only 3B active parameters per token. Running with Distributed Llama, Qwen reaches 13.04 tokens/s during generation on the same cluster. That is about 18 times faster.

The main architectural difference in Diffusion Gemma is that each forward pass processes a set of token candidates called a canvas. The model works on a single canvas over multiple forward passes in a process called denoising. To generate one canvas containing 256 tokens, it must therefore run multiple forward passes. The DiffusionGemma documentation recommends a maximum of 48 denoising steps and says that adaptive stopping typically completes a canvas in 12-16 steps.

Model	Synchronization data per token
Diffusion Gemma Q40 (14 denoising steps)	22.9 MB
Qwen3-30B-A3B Q40	1.6 MB

This causes multiple problems for distributed inference. Each synchronization exchanges state not just for a single candidate for the next token, but for the data needed to predict the next 256 tokens. During generation, Diffusion Gemma Q40 requires 22.9 MB of synchronization data per token (with 14 denoising steps), compared with just 1.6 MB per token for Qwen3-30B-A3B Q40.

Additionally, the model requires much more computation per token across all forward passes. According to an estimate by GPT-6 Astra, Diffusion Gemma requires 114 GFLOPs per token, compared with 7.6 GFLOPs per token for the autoregressive Gemma model. That is 15 times as much computation per token. Autoregressive models tend to be limited by memory bandwidth on GPUs, leaving some compute capacity unused. Diffusion Gemma takes advantage of this capacity, which is why it can be faster on GPUs, but not necessarily in the setup tested here.

Unfortunately, these results suggest that diffusion LLMs are not a good fit for distributed inference when nodes must synchronize over a slow network link. This is especially true when the goal is to combine low-end devices into a cluster to accelerate inference. Autoregressive models are not perfect either, but they remain the better option for this kind of setup.

You can try DDLLM yourself. The inference engine includes optimized CPU backends for ARM and AVX2, as well as support for Vulkan-compatible GPUs.