Sparse Computation

Sparse computation reduces the number of token or block computations during DLLM denoising while keeping the same evaluation protocol.

Supported Methods

Method	Paper	Config
SparseD	Paper	generation=vanilla generation.sparsed=true
DPad	Paper	generation=dpad
Sparse-dLLM	Paper	generation=sparse

Quick Start

Run all sparse-computation examples on LLaDA and GSM8K:

bash scripts/run_sparse_computation.sh

Method Examples

The examples below use model=llada-inst and dataset.name=gsm8k. To run the same commands with Dream, use model=dream-inst and set DREAM_INST_PATH; to switch benchmarks, replace dataset.name=gsm8k with any task available through lm-eval or the local tasks/ directory. Example tasks include humaneval_instruct, math-500, mbpp_instruct, ifeval, gpqa_main_generative_n_shot, and longbench.

SparseD

accelerate launch \
    --num_machines 1 \
    --num_processes 1 \
    eval.py \
    dataset.name=gsm8k \
    batch_size=1 \
    seed=1234 \
    attn_implementation=sdpa \
    flash_attention=true \
    generation=vanilla \
    generation.sparsed=true \
    generation.sparsed_select=0.5 \
    generation.sparsed_skip=0.2 \
    generation.sparsed_block_size=32 \
    generation.block_length=32 \
    generation.gen_length=256 \
    generation.steps=256 \
    model=llada-inst \
    hydra.run.dir=./outputs/examples/sparse_computation/llada-inst-sparsed-gsm8k

Note: Defaults are generation.sparsed_select=0.5, generation.sparsed_skip=0.2, and generation.sparsed_block_size=32; sparsed_select controls the selected sparse-token ratio, sparsed_skip controls the skipped-token ratio, and sparsed_block_size sets the sparse block size. For long-context settings, use generation.sparsed_select=0.3 and generation.sparsed_block_size=128.

DPad

accelerate launch \
    --num_machines 1 \
    --num_processes 1 \
    eval.py \
    dataset.name=gsm8k \
    batch_size=1 \
    seed=1234 \
    attn_implementation=sdpa \
    flash_attention=true \
    generation=dpad \
    generation.dropout=gaussian \
    generation.remasking=low_confidence \
    generation.early_termination=true \
    generation.block_length=32 \
    generation.gen_length=256 \
    generation.steps=256 \
    model=llada-inst \
    hydra.run.dir=./outputs/examples/sparse_computation/llada-inst-dpad-gsm8k

Note: Default is generation.dropout=gaussian; dropout chooses the sparse dropout pattern.

Sparse-dLLM

accelerate launch \
    --num_machines 1 \
    --num_processes 1 \
    eval.py \
    dataset.name=gsm8k \
    batch_size=1 \
    seed=1234 \
    attn_implementation=sdpa \
    flash_attention=true \
    generation=sparse \
    generation.keep_ratio=0.5 \
    generation.kernel_size=3 \
    generation.remasking=low_confidence \
    generation.early_termination=true \
    generation.block_length=32 \
    generation.gen_length=256 \
    generation.steps=256 \
    model=llada-inst \
    hydra.run.dir=./outputs/examples/sparse_computation/llada-inst-sparse-dllm-gsm8k

Note: Defaults are generation.keep_ratio=0.5 and generation.kernel_size=3; keep_ratio is the retained-token ratio for sparse updates, and kernel_size controls the local smoothing window.