Unquantized Safetensors (multi-GPU)

Efficient, easy-to-use platform for inference and serving local LLMs including an OpenAI compatible API server.
English | 简体中文

✨ Why Candle-vLLM?

🚀 Quick Start

📦 Install

Option 1 — One-line install (DEB or binary)

curl -sSL https://ericlbuehler.github.io/candle-vllm/install.sh | bash

Option 2 — Build from source

git clone git@github.com:EricLBuehler/candle-vllm.git
cd candle-vllm

# CUDA (11+, 12+, 13.0) — remove flashinfer,cutlass for sm_70/sm_75
cargo install --features cuda,nccl,flashinfer,cutlass --path .

# macOS/Metal
cargo install --features metal --path .

Option 3 — Docker

# Pass custom SM version and CUDA version: ./build_docker.sh "cuda,nccl,flashinfer,cutlass" sm_90 13.0.0
./build_docker.sh "cuda,nccl,flashinfer,cutlass"

▶️ Run

Using HuggingFace Model ID:

candle-vllm --m Qwen/Qwen3.6-27B-FP8 --ui-server

candle-vllm --m unsloth/Qwen3.5-122B-A10B-GGUF --f Q3_K_S --d 0,1 --ui-server

candle-vllm --m zai-org/GLM-5.2-FP8 --d 0,1,2,3,4,5,6,7 --ui-server

Using local model path:

# Local safetensors directory
candle-vllm --d 0,1,2,3,4,5,6,7 --m /home/data/GLM-5.2-FP8/ --ui-server

# Local GGUF file (single or split-shard)
candle-vllm --d 0,1 --m /home/data/model-Q4_K_M.gguf --ui-server

# Local directory containing GGUF files (auto-detected)
candle-vllm --d 0,1 --m /home/data/Qwen3.5-35B-A3B-GGUF/ --ui-server

Tip: Add --ui-server to launch the built-in ChatGPT-style Web UI. The UI server uses the API port minus one (e.g., API on 2000, UI on 1999).

📈 Performance

Single-request decode speed (input 4k, output 1k, on Hopper 80G)

🧠 Features

• OpenAI compatible API server for serving LLMs
• Streaming support in generation
• Efficient KV cache management with PagedAttention
• Continuous batching (batched decoding for incoming requests over time)
• In-situ quantization (and In-situ Marlin format conversion)
• GPTQ/Marlin format quantization (4-bit)
• Support Mac/Metal devices
• Support Multi-GPU inference (both multi-process and multi-threaded mode)
• Support Multi-node inference via TCP-based coordination
• Support Chunked Prefilling (default chunk size 8K)
• Support CUDA Graph
• Support Qwen3.5 MTP speculative decoding with CUDA Graph via --mtp
• Support Model Context Protocol (MCP) and OpenAI-compatible tool calling
• Support Prefix Caching
• Support Block-wise FP8 Models (SM90+, Qwen3 Series)
• Support FP8 KV Cache on all CUDA and Metal platforms
• Support TurboQuant KV Cache (turbo8/turbo4/turbo3) with native flash attention kernels
• Support Flashinfer Backend
• Support manual YaRN RoPE scaling override via --yarn-scaling-factor
• Support MXFP4/NVFP4 models
• Support GPTQ/AWQ/Marlin models, including pack-quantized AWQ MoE
• Support DeepSeek V3.2 and GLM-5.2 FP8 models

📘 Usage

Running Models

Tip: By default, candle-vllm starts an OpenAI-compatible API server at http://localhost:2000. Add --ui-server to also launch the built-in ChatGPT-style Web UI.

# FP8 model + Web UI
candle-vllm --m Qwen/Qwen3.6-27B-FP8 --ui-server

# GLM-5.2 FP8 model
candle-vllm --d 0,1,2,3,4,5,6,7 --m zai-org/GLM-5.2-FP8 --ui-server

# Unquantized Safetensors (multi-GPU)
candle-vllm --d 0,1 --w /home/Qwen3-30B-A3B-Instruct-2507/

# ISQ on-the-fly quantization
candle-vllm --m Qwen/Qwen3.6-27B --isq q4k

# FP4 Model
candle-vllm --m GadflyII/GLM-4.7-Flash-NVFP4 --ui-server

# GGUF model
candle-vllm --m unsloth/Qwen3-30B-A3B-Instruct-2507-GGUF --f Qwen3-30B-A3B-Instruct-2507-Q4_K_M.gguf --ui-server

# Manual YaRN scaling
candle-vllm --m Qwen/Qwen3.6-35B-A3B --yarn-scaling-factor 4.0 --ui-server

# FP8 Model (block-wise quant, build with cutlass feature)
candle-vllm --m Qwen/Qwen3.6-27B-FP8 --ui-server

# Faster GDN prefill on Hopper with slight precision loss
SM90_LOWER_PRECISION_GDN_PREFILL=1 candle-vllm --m Qwen/Qwen3.5-35B-A3B-FP8

# Qwen3.5 MTP speculative decoding (2 draft tokens per step)
candle-vllm --w /data/Qwen3.5-35B-A3B-FP8/ --mtp 2 --ui-server

# GLM-5.2 FP8 Model
candle-vllm --d 0,1,2,3,4,5,6,7 --m zai-org/GLM-5.2-FP8 --ui-server

# FP8 on MacOS/Metal (Dense)
candle-vllm --m Qwen/Qwen3-4B-Instruct-2507-FP8 --ui-server

# FP4 Model (MXFP4/NVFP4, MLX quantized format not supported)
candle-vllm --m GadflyII/GLM-4.7-Flash-NVFP4 --ui-server

# MXFP4
candle-vllm --m nm-testing/Qwen3-30B-A3B-MXFP4A16 --ui-server

# Local GGUF file via --m (recommended)
candle-vllm --m /home/data/Qwen3-30B-A3B-Instruct-2507-Q4_K_M.gguf --ui-server

# Local GGUF file via --f (legacy)
candle-vllm --f /home/data/Qwen3-30B-A3B-Instruct-2507-Q4_K_M.gguf --ui-server

# Local directory containing GGUF (auto-detected, mmproj loaded on demand)
candle-vllm --m /home/data/Qwen3.5-35B-A3B-GGUF/ --ui-server

# From HuggingFace (exact file)
candle-vllm --m unsloth/Qwen3-30B-A3B-Instruct-2507-GGUF --f Qwen3-30B-A3B-Instruct-2507-Q4_K_M.gguf --ui-server

# From HuggingFace (subfolder — downloads all GGUF files in the remote path)
candle-vllm --m unsloth/Qwen3.5-122B-A10B-GGUF --f Q3_K_S --d 0,1 --ui-server

# Local multi-shard GGUF (split files auto-discovered from local path)
candle-vllm --m /home/data/model-00001-of-00003.gguf --d 0,1 --ui-server

# GGUF on Apple Silicon
candle-vllm --m /home/qwq-32b-q4_k_m.gguf --ui-server
candle-vllm --m Qwen/QwQ-32B-GGUF --f qwq-32b-q4_k_m.gguf --ui-server

candle-vllm --m Qwen/Qwen3.6-27B --isq q4k

# Marlin-compatible GPTQ (4-bit, 128-group, desc_act=False)
candle-vllm --m thesven/Llama-3-8B-GPTQ-4bit

# Convert uncompressed model to Marlin-compatible format
python3 examples/convert_marlin.py --src /home/DeepSeek-R1-Distill-Qwen-14B/ --dst /home/DeepSeek-R1-Distill-Qwen-14B-GPTQ_4bit-128g
candle-vllm --w /home/DeepSeek-R1-Distill-Qwen-14B-GPTQ_4bit-128g

# Convert AWQ to Marlin-compatible format
python3 examples/convert_awq_marlin.py --src /home/Meta-Llama-3.1-8B-Instruct-AWQ-INT4/ --dst /home/Meta-Llama-3.1-8B-Instruct-AWQ-INT4-Marlin/ --bits 4 --method awq --group 128 --nk False
candle-vllm --d 0 --w /home/Meta-Llama-3.1-8B-Instruct-AWQ-INT4-Marlin/

# AWQ MoE (pack-quantized), e.g. Qwen3-Coder-Next
candle-vllm --m cyankiwi/Qwen3-Coder-Next-AWQ-4bit

# Direct Marlin-format model
candle-vllm --w /home/DeepSeek-R1-Distill-Qwen-14B-GPTQ-Marlin/

🗜️ TurboQuant KV Cache

TurboQuant compresses the KV cache using Walsh-Hadamard transform for higher throughput and longer context:

# Turbo4 (4-bit KV cache, ~3.7x compression)
candle-vllm --w /data/Qwen3.5-27B-FP8/ --kvcache-dtype turbo4

# Turbo8 (FP8 K + 4-bit V, ~2.6x compression)
candle-vllm --w /data/Qwen3.5-27B-FP8/ --kvcache-dtype turbo8

# Turbo3 (3-bit K + 4-bit V, ~4.7x compression)
candle-vllm --w /data/Qwen3.5-27B-FP8/ --kvcache-dtype turbo3

# FP8 KV Cache
candle-vllm --w /data/Qwen3.5-35B-A3B-FP8/ --kvcache-dtype fp8

Note: TurboQuant uses native flash attention kernels (flashinfer is automatically disabled). Supported on both CUDA (SM70+) and Metal (Apple Silicon) platforms. MLA models (DeepSeek, GLM4/GLM-5.2) auto-fallback to standard KV cache as TurboQuant is incompatible with their compressed KV layout.

🖥️ Multi-GPU Inference

# QwQ-32B BF16 on two GPUs
candle-vllm --d 0,1 --w /home/QwQ-32B/

# QwQ-32B 4-bit AWQ on two GPUs
python3 examples/convert_awq_marlin.py --src /home/QwQ-32B-AWQ/ --dst /home/QwQ-32B-AWQ-Marlin/ --bits 4 --method awq --group 128 --nk False
candle-vllm --d 0,1 --w /home/QwQ-32B-AWQ-Marlin/

# Add --multithread parameter
candle-vllm --multithread --d 0,1 --w /home/QwQ-32B/

# Troubleshooting
export NCCL_P2P_DISABLE=1  # disable P2P if encountering illegal memory access

🌐 Multi-Node Inference

Distribute inference across multiple machines using TCP-based NCCL bootstrap. No MPI required.

# On master node (192.168.1.100):
candle-vllm --d 0,1,2,3,4,5,6,7 --w /data/DeepSeek-R1-AWQ-Marlin/ \
  --num-nodes 2 --node-rank 0 --master-addr 192.168.1.100 --master-port 29500

# On worker node (192.168.1.101):
candle-vllm --d 0,1,2,3,4,5,6,7 --w /data/DeepSeek-R1-AWQ-Marlin/ \
  --num-nodes 2 --node-rank 1 --master-addr 192.168.1.100 --master-port 29500

All nodes must have model weights locally and be TCP-reachable on --master-port (default 29500).

📐 NUMA Binding

sudo apt-get install numactl

# 8 GPUs, 2 NUMA nodes
MAP_NUMA_NODE=0,0,0,0,1,1,1,1 numactl --cpunodebind=0 --membind=0 candle-vllm --d 0,1,2,3,4,5,6,7 --w /home/data/DeepSeek-V2-Chat-AWQ-Marlin

# 4 GPUs
MAP_NUMA_NODE=0,0,0,0 numactl --cpunodebind=0 --membind=0 candle-vllm --d 0,1,2,3 --w /home/data/DeepSeek-V2-Chat-AWQ-Marlin

⚙️ CLI Reference

Binding examples:

candle-vllm --h 127.0.0.1 --p 8000 --m Qwen/Qwen3.6-27B-FP8
candle-vllm --h 127.0.0.1:8000 --m Qwen/Qwen3.6-27B-FP8
candle-vllm --h '[::1]:8000' --m Qwen/Qwen3.6-27B-FP8
candle-vllm --h unix:///tmp/candle-vllm.sock --m Qwen/Qwen3.6-27B-FP8

📚 Documentation

Using Agents under Candle-vLLM backend: xbot · OpenCode · Kilo Code

🛠️ Roadmap

• OpenAI-compatible API server (streaming)
• Continuous batching
• Flash Attention (CUDA)
• FlashInfer backend
• CUDA Graph
• Chunked Prefill
• Prefix Caching (CUDA & Metal)
• Multi-GPU inference (multi-process & multi-threaded)
• Multi-node tensor parallelism (TCP-based NCCL, no MPI)
• In-situ quantization (GGML/GGUF + Marlin)
• FP8 KV Cache (CUDA & Metal, all backends)
• TurboQuant KV Cache (2–4 bit compression)
• FP8 Models (block-wise, SM90+)
• MXFP4/NVFP4 Model Support
• DeepSeek V3.2 and GLM-5.2 FP8 Model Support
• MCP Integration & Tool Calling
• Built-in ChatGPT-style Web UI

📚 References

• Python implementation: vllm-project
• vllm paper

Report Issue

If you encounter any problems, please create an issue.