Fast-dLLM v1: Training-free Acceleration of Diffusion LLM

 

Fast-dLLM v1 is a training-free inference acceleration framework for diffusion-based Large Language Models (dLLMs). It supports efficient inference for models like Dream and LLaDA by enabling KV Cache and Parallel Decoding.

Key Features

1. Key-Value Cache for Block-Wise Decoding We propose an efficient block-wise decoding KV Cache mechanism for Masked Diffusion Models (MDMs). By reusing attention Key-Value activations across multiple steps within each block, our approach avoids redundant computation and significantly accelerates inference. Furthermore, our DualCache extension also caches masked suffix tokens, enabling even greater speedup with negligible accuracy loss.

2. Confidence-Aware Parallel Decoding Instead of decoding tokens sequentially, we introduce a confidence-aware parallel decoding scheme. At each step, only tokens with confidence over a threshold are unmasked in parallel, while uncertain ones remain masked for future steps. This selective approach effectively balances decoding efficiency and output quality.

3. Overall Performance Overall, introducing the KV Cache mechanism yields significant speed improvements for all tasks and sequence lengths, typically achieving a 2x to 3.6x speedup compared to the vanilla backbone. When the parallel decoding strategy is applied individually, we see additional acceleration, often pushing speedups to 4x-6x for the evaluated settings, particularly as the generation length increases.

Demo

https://github.com/user-attachments/assets/32bbff97-6e60-4e14-95c0-2cbec136476f

File Structure

v1/
├── README.md               # This file
├── requirements.txt        # Dependencies for inference & evaluation
├── dream/                  # Dream model related code
│   ├── model/              # Dream model definition
│   ├── eval.py             # Evaluation harness integration
│   ├── eval.md             # Evaluation guide
│   ├── eval_gsm8k.sh       # GSM8K evaluation script
│   ├── eval_humaneval.sh   # HumanEval evaluation script
│   └── demo_multiturn_chat.py  # Multi-turn chat demo
└── llada/                  # LLaDA model related code
    ├── model/              # LLaDA model definition
    ├── generate.py         # Core generation with cache & parallel decoding
    ├── eval_llada.py       # Evaluation harness integration
    ├── eval.md             # Evaluation guide
    ├── eval_gsm8k.sh       # GSM8K evaluation script
    ├── eval_humaneval.sh   # HumanEval evaluation script
    ├── chat.py             # Command-line chat interface
    └── app.py              # Gradio web demo

Installation

cd v1
pip install -r requirements.txt

Usage

1. Using LLaDA Model

Interactive Chat

python llada/chat.py --gen_length 128 --steps 128 --block_size 32

Parameter descriptions:

• --gen_length: Maximum length of generated text
• --steps: Number of sampling steps
• --block_size: Cache block size
• --use_cache: Whether to use cache
• --if_cache_position: Whether to use dual cache
• --threshold: Confidence threshold

Web Demo

pip install gradio
cd llada
python app.py

Model Evaluation

Benchmark	Gen Length	LLaDA	+Cache	+Parallel	+Cache+Parallel (Fast-dLLM)
GSM8K (5-shot)	256	79.3 6.73 (1×)	79.5 21.23 (3.2×)	79.2 16.53 (2.5×)	78.5 54.4 (8.1×)
	512	77.5 3.23 (1×)	77.0 10.43 (3.3×)	77.6 18.63 (5.8×)	77.2 35.3 (11.0×)
HumanEval (0-shot)	256	41.5 30.5 (1×)	42.7 40.73 (1.3×)	43.9 101.53 (3.3×)	43.3 114.1 (3.7×)
	512	43.9 18.4 (1×)	45.7 29.33 (1.6×)	43.3 57.13 (3.1×)	44.5 73.7 (4.0×)

Each cell presents the accuracy (top row, in percentage) and the decoding throughput (middle row, in tokens per second) with relative speedup (bottom row) to the LLaDA baseline.

For detailed evaluation instructions, please refer to:

• LLaDA Evaluation Guide
• Dream Evaluation Guide

2. Using Dream Model

For detailed evaluation instructions on GSM8K and HumanEval benchmarks, please refer to Dream Evaluation Guide.

Citation

@misc{wu2025fastdllmtrainingfreeaccelerationdiffusion,
      title={Fast-dLLM: Training-free Acceleration of Diffusion LLM by Enabling KV Cache and Parallel Decoding}, 
      author={Chengyue Wu and Hao Zhang and Shuchen Xue and Zhijian Liu and Shizhe Diao and Ligeng Zhu and Ping Luo and Song Han and Enze Xie},
      year={2025},
      eprint={2505.22618},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2505.22618}, 
}

Acknowledgements

We would like to thank the authors of LLaDA and Dream for their excellent work and open-source contributions.