Integrated DLLM Setups

Integrated settings combine multiple DLLM acceleration choices and compare them against integrated model-level acceleration.

Example Setups

This section shows representative combinations used by the examples; the same acceleration components can be applied to other supported DLLM model configs when compatible.

Setup	Paper	Config
Prefix Cache + Confidence	/	cache=prefix generation=vanilla generation.threshold=0.9
dKVCache + Confidence	/	cache=dkvcache generation=vanilla generation.threshold=0.9
SparseD + Confidence	/	generation=vanilla generation.sparsed=true generation.threshold=0.9
dParallel + Confidence	/	model=dparallel_llada-inst generation=vanilla generation.threshold=0.9
Fast-dLLM v2	Paper	model=fast_dllm_v2_7b-inst generation=fast_dllm_v2 generation.threshold=0.9

Quick Start

Run all integrated examples on GSM8K:

bash scripts/run_integrated.sh

Method Examples

The examples below use model=llada-inst and dataset.name=gsm8k, except where a setup requires its own model config, such as dParallel or Fast-dLLM v2. For standard LLaDA/Dream-compatible setups, use model=dream-inst and set DREAM_INST_PATH; for dParallel on Dream, use model=dparallel_dream-inst and set DPARALLEL_DREAM_INST_PATH. To switch benchmarks, replace dataset.name=gsm8k with any task available through lm-eval or the local tasks/ directory. Example tasks include humaneval_instruct, math-500, mbpp_instruct, ifeval, gpqa_main_generative_n_shot, and longbench.

LLaDA + Prefix Cache + Confidence

accelerate launch \
    --num_machines 1 \
    --num_processes 1 \
    eval.py \
    dataset.name=gsm8k \
    batch_size=1 \
    seed=1234 \
    attn_implementation=sdpa \
    flash_attention=true \
    cache=prefix \
    generation=vanilla \
    generation.block_length=32 \
    generation.threshold=0.9 \
    generation.gen_length=128 \
    generation.steps=128 \
    model=llada-inst \
    hydra.run.dir=./outputs/examples/integrated/llada-inst-prefixcache-confidence-gsm8k

Note: This setup combines cache=prefix with generation.threshold=0.9; Prefix Cache reuses prefix-side KV states, and the confidence threshold accepts tokens whose confidence is greater than or equal to 0.9.

LLaDA + dKVCache + Confidence

accelerate launch \
    --num_machines 1 \
    --num_processes 1 \
    eval.py \
    dataset.name=gsm8k \
    batch_size=1 \
    seed=1234 \
    attn_implementation=sdpa \
    flash_attention=true \
    cache=dkvcache \
    generation=vanilla \
    generation.block_length=32 \
    generation.threshold=0.9 \
    generation.cache_reloading_step=4 \
    generation.gen_length=128 \
    generation.steps=128 \
    model=llada-inst \
    hydra.run.dir=./outputs/examples/integrated/llada-inst-dkvcache-confidence-gsm8k

Note: This setup combines cache=dkvcache, generation.cache_reloading_step=4, and generation.threshold=0.9; cache_reloading_step refreshes the KV cache every four decoding steps, and the confidence threshold accepts tokens whose confidence is greater than or equal to 0.9.

LLaDA + SparseD + Confidence

accelerate launch \
    --num_machines 1 \
    --num_processes 1 \
    eval.py \
    dataset.name=gsm8k \
    batch_size=1 \
    seed=1234 \
    attn_implementation=sdpa \
    flash_attention=true \
    generation=vanilla \
    generation.sparsed=true \
    generation.sparsed_select=0.5 \
    generation.sparsed_skip=0.2 \
    generation.sparsed_block_size=32 \
    generation.block_length=32 \
    generation.threshold=0.9 \
    generation.gen_length=128 \
    generation.steps=128 \
    model=llada-inst \
    hydra.run.dir=./outputs/examples/integrated/llada-inst-sparsed-confidence-gsm8k

Note: This setup combines SparseD defaults generation.sparsed_select=0.5, generation.sparsed_skip=0.2, and generation.sparsed_block_size=32 with generation.threshold=0.9; SparseD reduces token computation, and the confidence threshold accepts tokens whose confidence is greater than or equal to 0.9.

LLaDA + dParallel + Confidence

accelerate launch \
    --num_machines 1 \
    --num_processes 1 \
    eval.py \
    dataset.name=gsm8k \
    batch_size=1 \
    seed=1234 \
    attn_implementation=sdpa \
    flash_attention=true \
    generation=vanilla \
    generation.block_length=32 \
    generation.threshold=0.9 \
    generation.gen_length=128 \
    generation.steps=128 \
    model=dparallel_llada-inst \
    hydra.run.dir=./outputs/examples/integrated/dparallel-llada-inst-confidence-gsm8k

Note: This setup uses model=dparallel_llada-inst with generation.threshold=0.9; dParallel changes the model wrapper, and the confidence threshold keeps the same token-acceptance rule.

Fast-dLLM v2

accelerate launch \
    --num_machines 1 \
    --num_processes 1 \
    eval.py \
    dataset.name=gsm8k \
    batch_size=1 \
    seed=1234 \
    attn_implementation=sdpa \
    flash_attention=true \
    generation=fast_dllm_v2 \
    generation.block_length=32 \
    generation.threshold=0.9 \
    generation.gen_length=2048 \
    generation.steps=2048 \
    generation.small_block_size=8 \
    add_bos_token=false \
    model.generation.add_bos_token=false \
    model=fast_dllm_v2_7b-inst \
    hydra.run.dir=./outputs/examples/integrated/fastdllmv2-gsm8k

Note: This setup uses model=fast_dllm_v2_7b-inst, generation.block_length=32, generation.threshold=0.9, and generation.small_block_size=8; Fast-dLLM v2 uses its own block-diffusion generation path, and small_block_size controls the sub-block granularity.