DeepSeek V4
August 31, 2026 · View on GitHub
DeepSeek-V4 is the next-generation Mixture-of-Experts language model from DeepSeek-AI. It extends the V3 design with Hyper-Connections (mHC) for multi-stream residual mixing, Compressed Sparse Attention (CSA) with a learned token-importance indexer (DSA), hash-routed MoE layers for the first few decoder blocks, and a refined Multi-Token Prediction (MTP) head with separate e_proj / h_proj projections.
DeepSeek V4 models are supported via the Bridge system with auto-detected configuration and weight mapping.
Verified configurations
Choose an exact recorded configuration to see its command and expected result. These selectors are generated from the authoritative verification cards and never synthesize combinations.
Run a configuration
Choose a workflow, precision, and exact recorded combination. The command and expected result update below.
Import · CPU
○ UnverifiedExact command
./scripts/conversion/convert.sh import --executor slurm --device cpu --nodes 1 --hf-model deepseek-ai/DeepSeek-V4-Flash --hf-revision 60d8d70770c6776ff598c94bb586a859a38244f1 --megatron-path work/model-verification/dsv4-flash/import-cpu --torch-dtype bfloat16 --trust-remote-codeExpected result
CPU import requires approximately 570 GB of CPU RAM for the full BF16 weight materialisation (285B parameters x 2 bytes). Verification requires a high-memory node with enough additional headroom for conversion workspace and strict checkpoint reload.
Import · GPU
✓ VerifiedExact command
./scripts/conversion/convert.sh import --executor slurm --device gpu --nodes 1 --gpus-per-node 4 --ep 4 --hf-model deepseek-ai/DeepSeek-V4-Flash --hf-revision 60d8d70770c6776ff598c94bb586a859a38244f1 --megatron-path work/model-verification/dsv4-flash/import-gpu --torch-dtype bfloat16 --trust-remote-codeExpected result
The command exits successfully and creates a reloadable iter_0000000 Megatron checkpoint.
Export · CPU
○ UnverifiedExact command
./scripts/conversion/convert.sh export --executor slurm --device cpu --nodes 1 --hf-model deepseek-ai/DeepSeek-V4-Flash --hf-revision 60d8d70770c6776ff598c94bb586a859a38244f1 --megatron-path work/model-verification/dsv4-flash/import-cpu/iter_0000000 --hf-path work/model-verification/dsv4-flash/export-cpu --torch-dtype bfloat16 --trust-remote-codeExpected result
CPU export requires a high-memory node with enough headroom for the full BF16 weight set and tensor-merging workspace, followed by strict Hugging Face reload.
Export · GPU
✓ VerifiedExact command
./scripts/conversion/convert.sh export --executor slurm --device gpu --nodes 2 --gpus-per-node 4 --tp 1 --pp 1 --ep 8 --etp 1 --hf-model deepseek-ai/DeepSeek-V4-Flash --hf-revision 60d8d70770c6776ff598c94bb586a859a38244f1 --megatron-path work/model-verification/dsv4-flash/import-gpu/iter_0000000 --hf-path work/model-verification/dsv4-flash/export-gpu --torch-dtype bfloat16 --export-weight-dtype bfloat16 --trust-remote-code --not-strictExpected result
Two GB200 nodes produced 46 complete safetensors shards containing all 35,020 expected non-scale source keys. BF16 export intentionally omitted 34,167 quantization-scale companions; 35,017 exported tensors are BF16 and the three I32 tid2eid routing tables exactly match their source values after the expected integer cast. Two independent Transformers processes each strictly reloaded all 1,500 model modules across four GPUs with no CPU or disk placement and produced the same bounded greedy token. --not-strict permits only the intentional scale omission; exact key, shard, dtype, routing-table value, and reload checks remain required correctness gates.
Pretrain · GB200
✓ VerifiedRecorded metrics
- Initial loss
- 7.250204
- Final loss
- 3.280722
- Step time · last 10 avg
- 7,856.370 ms
- Model throughput · last 10 avg
- 189.690 TFLOP/s/GPU
- Token throughput · last 10 avg
- 2,085.441 tokens/s/GPU
Exact command
./scripts/training/train.sh --nodes 16 --gpus-per-node 4 --recipe deepseek_v4_flash_pretrain_64gpu_gb200_fp8mx_library_config --mode pretrain --dataset megatron-indexed --max_steps 100 --pretrained_checkpoint work/model-verification/dsv4-flash/import-gpu/iter_0000000 'dataset.blend=[["work/data/the-pile/my-gpt3_08_text_document"],null]' dataset.path_to_cache=work/cache/the-pile dataset.num_workers=0 dataset.random_seed=1234 rng.seed=1234 scheduler.lr_warmup_iters=10 scheduler.lr_decay_iters=100 validation.eval_interval=0 validation.eval_iters=0 ddp.check_for_nan_in_grad=true ddp.check_for_large_grads=true rerun_state_machine.check_for_nan_in_loss=true checkpoint.exit_on_missing_checkpoint=false checkpoint.finetune=false checkpoint.load=null checkpoint.load_optim=false checkpoint.load_rng=false checkpoint.save_optim=true checkpoint.save_rng=true checkpoint.async_save=false --save_dir work/model-verification/dsv4-flash/pretrain-ref --save_interval 50 logger.log_interval=1 logger.log_throughput=true logger.tensorboard_dir=null dist.distributed_timeout_minutes=120Expected result
On 64 GB200 (16 nodes x 4 GPUs), TP1/PP4/VPP4/EP16/CP1 uses dense DP16, expert DP1, and GBS/MBS 256/1. The command loads the imported model weights while starting optimizer and RNG state fresh. It uses HybridEP natural routing, grouped GEMM, TE fused cross entropy, selective recompute over mhc and mla_up_proj, and MXFP8 parameter gather and gradient-buffer reuse without activation offload. Expert capacity, paged stash, CUDA graphs, and forced load balancing remain disabled. The run completed 100 finite LM/MTP-loss steps with no skipped or NaN iterations and wrote complete grouped-MXFP8 checkpoints containing model, optimizer, scheduler, data-order, and RNG state at steps 50 and 100. The item-specific Bridge revision pins the compatible Megatron-LM dev revision used by this workload without changing the card's default environment for other verification items.
SFT · GB200
✓ VerifiedRecorded metrics
- Initial loss
- 0.9435847
- Final loss
- 0.2834835
- Step time · last 10 avg
- 8,575.850 ms
- Model throughput · last 10 avg
- 42.180 TFLOP/s/GPU
- Token throughput · last 10 avg
- 477.620 tokens/s/GPU
- Peak allocated memory
- 184.960 GiB
- Peak reserved memory
- 187.040 GiB
Exact command
./scripts/training/train.sh --nodes 8 --gpus-per-node 4 --recipe deepseek_v4_flash_sft_openmath_thinking_packed_gb200_config --mode sft --step-func dsv4_step --pretrained_checkpoint work/models/deepseek-v4-flash-megatron --save_dir work/model-verification/dsv4-flash/sft-ref --save_interval 100 --max_steps 100 --seq_length 1024 --pipeline_model_parallel_size 4 --context_parallel_size 1 --expert_model_parallel_size 8 'model.pipeline_model_parallel_layout=Et*11|t*11|t*11|t*10mL' scheduler.lr_warmup_iters=10 scheduler.lr_decay_iters=100 validation.eval_interval=0 validation.eval_iters=0 rng.seed=5678 ddp.check_for_nan_in_grad=true ddp.check_for_large_grads=true rerun_state_machine.check_for_nan_in_loss=true checkpoint.exit_on_missing_checkpoint=false checkpoint.load=null checkpoint.load_optim=false checkpoint.load_rng=false checkpoint.save_optim=false ddp.overlap_grad_reduce=false logger.log_interval=1 logger.log_throughput=true logger.tensorboard_dir=null dist.distributed_timeout_minutes=180Expected result
On 32 GB200 (8 nodes x 4 GPUs), TP1/PP4/EP8/CP1 with dense DP8 and expert DP1, GBS/MBS 128/1. OpenMathInstruct-2 thinking data uses offline packing at seq_length=1024 with fixed token and cumulative-boundary shapes. The GB200 recipe uses selective recompute over moe, mhc, mla_up_proj, and layernorm, attention activation offload, HybridEP dispatch, and DSA kernel fusion without enabling the optional DSA indexer loss. The run completed 100 steps with finite LM and MTP losses, no skipped or NaN iterations, and a complete model checkpoint at step 100. A fresh process reloaded that checkpoint and completed finite step 101. moe_grouped_gemm=True is a recipe default. dist.distributed_timeout_minutes must be 180 or above to allow offline data packing (approximately 63 minutes) before training begins.
Long Context · GB200
✓ VerifiedRecorded metrics
- Initial loss
- 0.94383
- Final loss
- 0.2837
- Step time · last 10 avg
- 13,806.500 ms
- Model throughput · last 10 avg
- 13.620 TFLOP/s/GPU
- Token throughput · last 10 avg
- 148.336 tokens/s/GPU
Exact command
./scripts/training/train.sh --nodes 16 --gpus-per-node 4 --recipe deepseek_v4_flash_sft_openmath_thinking_packed_config --step-func dsv4_step --pretrained_checkpoint work/models/deepseek-v4-flash-megatron --save_dir work/model-verification/dsv4-flash/sft-long-context-ref --save_interval 100 --max_steps 100 model.pipeline_model_parallel_size=4 'model.pipeline_model_parallel_layout=Et*11|t*11|t*11|t*10mL' model.context_parallel_size=2 model.cp_partition_mode=contiguous model.expert_model_parallel_size=8 model.moe_grouped_gemm=false model.recompute_granularity=full model.recompute_method=uniform model.recompute_num_layers=1 dataset.seq_length=1024 scheduler.lr_warmup_iters=10 scheduler.lr_decay_iters=100 validation.eval_interval=0 validation.eval_iters=0 rng.seed=5678 ddp.check_for_nan_in_grad=true ddp.check_for_large_grads=true rerun_state_machine.check_for_nan_in_loss=true checkpoint.exit_on_missing_checkpoint=false checkpoint.load=null ddp.overlap_grad_reduce=false logger.log_interval=1 logger.log_throughput=true logger.tensorboard_dir=null dist.distributed_timeout_minutes=180Expected result
CP=2 with contiguous partitioning and offline-packed OpenMathInstruct-2 at seq_length=1024 demonstrates sequence packing and context parallelism working together over 100 training steps with finite loss and no skipped or NaN iterations.
LoRA · GB200
✓ VerifiedRecorded metrics
- Initial loss
- 0.9435847
- Final loss
- 0.2914867
- Step time · last 10 avg
- 6,944.330 ms
- Model throughput · last 10 avg
- 52.000 TFLOP/s/GPU
- Token throughput · last 10 avg
- 589.834 tokens/s/GPU
- Peak allocated memory
- 53.712 GiB
- Peak reserved memory
- 53.953 GiB
Exact command
./scripts/training/train.sh --nodes 8 --gpus-per-node 4 --recipe deepseek_v4_flash_peft_openmath_thinking_packed_gb200_config --mode lora --step-func dsv4_step --pretrained_checkpoint work/models/deepseek-v4-flash-megatron --save_dir work/model-verification/dsv4-flash/peft-ref --save_interval 100 --max_steps 100 --seq_length 1024 --pipeline_model_parallel_size 4 --context_parallel_size 1 --expert_model_parallel_size 8 'model.pipeline_model_parallel_layout=Et*11|t*11|t*11|t*10mL' scheduler.lr_warmup_iters=10 scheduler.lr_decay_iters=100 validation.eval_interval=0 validation.eval_iters=0 rng.seed=5678 ddp.check_for_nan_in_grad=true ddp.check_for_large_grads=true rerun_state_machine.check_for_nan_in_loss=true checkpoint.exit_on_missing_checkpoint=false checkpoint.load=null checkpoint.load_optim=false checkpoint.load_rng=false checkpoint.save_optim=false ddp.overlap_grad_reduce=false logger.log_interval=1 logger.log_throughput=true logger.tensorboard_dir=null dist.distributed_timeout_minutes=180Expected result
On 32 GB200 (8 nodes x 4 GPUs), TP1/PP4/EP8/CP1 with dense DP8 and expert DP1, GBS/MBS 128/1. The recipe applies rank-32, alpha-32 LoRA with zero dropout to linear_q_down_proj, linear_q_up_proj, linear_kv_proj, linear_proj, linear_fc1, and linear_fc2. Routed experts use separate per-expert adapters; shared experts are also adapted. The run preserves the packed OpenMath thinking data, objective, natural routing, seed, and warmup/decay horizon used by the verified SFT item while freezing the base model and using PEFT's own 1e-4 peak learning rate. It disables activation recompute and offload because the reduced PEFT training-state footprint fits with 53.712 GiB peak allocated memory. Transformer Engine CUDA graphs cover moe_router and moe_preprocess, while HybridEP uses 32 flex-dispatcher SMs, 8 ranks per NVLink domain, 128-token combine chunks, a 72-GPU domain, and MNNVL. All 100 steps completed with finite LM/MTP losses and zero skipped or NaN iterations. LM/MTP losses were 0.9435847/0.2250465 at step 1 and 0.2914867/0.02638751 at step 100. The adapter-only checkpoint reloaded after the same base checkpoint in a fresh process and completed finite step 101 with LM loss 0.2827368 and MTP loss 0.02594211.
Benchmark · GB200
✓ VerifiedRecorded metrics
- Initial loss
- 13.5875
- Final loss
- 3.62683
- Step time · last 10 avg
- 8,134.600 ms
- Model throughput · last 10 avg
- 731.400 TFLOP/s/GPU
- Token throughput · last 10 avg
- 8,056.450 tokens/s/GPU
Exact command
./scripts/training/train.sh --nodes 32 --gpus-per-node 4 --recipe deepseek_v4_flash_pretrain_128gpu_gb200_fp8mx_config --max_steps 50 scheduler.lr_warmup_iters=5 scheduler.lr_decay_iters=50 validation.eval_interval=0 validation.eval_iters=0 checkpoint.exit_on_missing_checkpoint=false logger.log_interval=1 logger.log_throughput=true logger.tensorboard_dir=null dist.distributed_timeout_minutes=120Expected result
On 128 GB200, PP1/EP64/CP1/TP1/DP2 with GBS/MBS 2048/1, FP8-MX, HybridEP dispatch, and full-iteration CUDA graphs completes 50 steps with finite losses, no skipped or NaN iterations, and all five metrics recorded. This benchmark uses forced routing, a static expert-rank capacity, paged stash, and a different batch/objective contract, so its metrics must not be compared with natural-routing library pretraining as convergence evidence.
Benchmark · GB300
✓ VerifiedRecorded metrics
- Initial loss
- 12.59616
- Final loss
- 3.160061
- Step time · last 10 avg
- 7,840.320 ms
- Model throughput · last 10 avg
- 759.280 TFLOP/s/GPU
- Token throughput · last 10 avg
- 8,358.842 tokens/s/GPU
Exact command
./scripts/training/train.sh --wait --nodes 32 --gpus-per-node 4 --recipe deepseek_v4_flash_pretrain_128gpu_gb300_fp8mx_config --mode pretrain --max_steps 50 --seq_length 4096 logger.save_config_filepath=work/model-verification/deepseek-v4-flash/gb300-performance/ConfigContainer.yamlExpected result
On exactly 128 GB300s, the canonical MXFP8 mock-data recipe completes exactly 50 optimizer steps at TP1/PP1/CP1/EP64/ETP1, GBS/MBS 2048/1, and sequence length 4096. All 50 keyed rows have finite loss with zero skipped or NaN iterations. Loss moves from 12.59616 to 3.160061; the final ten steps average 7840.320 ms, 759.280 TFLOP/s/GPU, and 8358.842 tokens/s/GPU. The resolved configuration persists.