ISQ on-the-fly quantization

Blazing-fast LLM inference in pure Rust. No PyTorch. No Python runtime. Just fast, portable, production-ready inference.
English | 简体中文

✨ Why xInfer?

📽️ Demo

🚀 Quick Start

📦 Install

Option 1 — Install DEB or Python package

curl -sSL https://guoqingbao.github.io/xinfer/install.sh | bash

Option 2 — npm

npm install -g xinfer-ai

▶️ Run

Using HuggingFace Model ID:

xinfer --m Qwen/Qwen3.6-27B-FP8 --kvcache-dtype turbo4 --ui-server

Using local model path:

xinfer --m /home/Qwen3.6-35B-A3B --d 0,1 --ui-server

Python usage:

# python3 -m xinfer.chat
python3 -m xinfer.server --m Qwen/Qwen3.6-27B-FP8 --kvcache-dtype turbo4 --ui-server

MTP

xinfer --w /home/Qwen3.6-35B-A3B --d 0,1 --ui-server --mtp 2

Tip: Open http://IP:8001 for the built-in chat UI, or use http://IP:8000/v1/ as your API Base URL.

🗜️ KV Cache Compression

Add --kvcache-dtype to compress KV cache and extend context length:

📈 Performance

Tested on V100-32G, A100-40G, Hopper-80G and RTX 5090

Full benchmarks →

🧠 Supported Models

• ✅ LLaMa (LLaMa2, LLaMa3, LLaMa4, IQuest-Coder)
• ✅ Qwen (Qwen2, Qwen3)
• ✅ Qwen2/Qwen3 MoE
• ✅ Qwen3 Next
• ✅ Qwen3.5/3.6 Dense/MoE (27B, 35B, 122B, 397B, Multimodal model)
• ✅ Mistral v1, v2
• ✅ Mistral-3-VL Reasoning (3B, 8B, 14B, Multimodal model)
• ✅ GLM4 (0414)
• ✅ GLM4 MoE (4.6/4.7)
• ✅ GLM4.7 Flash
• ✅ GLM 5.2 (DeepSeek V3.2 DSA architecture)
• ✅ DeepSeek V3/R1/V3.2
• ✅ Phi3 / Phi4 (Phi-3, Phi-4, Phi-4-mini, etc.)
• ✅ Gemma3/Gemma4 (Multimodal model)
• ✅ Qwen3-VL (Dense, Multimodal model)
• ✅ MiroThinker-v1.5 (30B, 235B)

Formats: Safetensors (BF16, FP8-blockwise, GPTQ, AWQ, MXFP4, NVFP4) | GGUF (all quant types) | ISQ (on-the-fly quantization)

TurboQuant KV Cache — Run 30B+ Models on Consumer GPUs

TurboQuant compresses KV cache to 2–4 bits via Walsh-Hadamard transform rotation + per-head absmax quantization. Max context tokens with turbo4:

Hybrid models (Qwen3.6) have fewer full attention layers, making TurboQuant especially effective. MLA models (DeepSeek, GLM4.7 Flash) use fp8 instead. The KV budget in the table is the theoretical maximum; actual usage can only utilize up to 90% of the KV budget (--kv-fraction 0.9), leaving room for runtime and batching buffers.

# 35B MoE on single 24/32 GB GPU
xinfer --m unsloth/Qwen3.6-35B-A3B-NVFP4 --kvcache-dtype turbo4

# Production precision
xinfer --m Qwen/Qwen3.6-35B-A3B-FP8 --kvcache-dtype fp8

# 27B Dense + turbo4
xinfer --m Qwen/Qwen3.6-27B-FP8 --kvcache-dtype turbo4

# 26B Gemma4 (local model, occupy more kvcache with --kv-fraction)
xinfer --w /data/gemma-4-26B-A4B-it --ui-server --port 9000 --kv-fraction 0.8

# 30B MoE GGUF + turbo4
xinfer --m unsloth/Qwen3-30B-A3B-Instruct-2507-GGUF \
  --f Qwen3-30B-A3B-Instruct-2507-Q4_K_M.gguf --kvcache-dtype turbo4

# Metal/MacOS
xinfer --m unsloth/Qwen3.5-4B-GGUF --f Qwen3.5-4B-Q4_K_M.gguf

📘 Usage

For Python installaion, running model with python3 -m xinfer.server

For Docker builds, refer to Run xInfer in Docker →

Running Models

Tip: By default, xInfer starts an OpenAI-compatible API server at http://localhost:8000. Add --ui-server to also launch the built-in ChatGPT-style Web UI at http://localhost:8001.

# FP8 model (sm90+ with cutlass) + web UI
xinfer --m Qwen/Qwen3.6-27B-FP8 --ui-server

# Unquantized Safetensors (multi-GPU)
xinfer --d 0,1 --m Qwen/Qwen3-30B-A3B-Instruct-2507 --kvcache-dtype fp8

# ISQ on-the-fly quantization
xinfer --m Qwen/Qwen3.6-35B-A3B --isq q4k

# NVFP4 model
xinfer --m unsloth/Qwen3.6-27B-NVFP4

# MXFP4
xinfer --m olka-fi/Qwen3.5-4B-MXFP4

# GGUF model (4-bit KvCache)
xinfer --m unsloth/Qwen3.5-27B-GGUF --f Qwen3.5-27B-Q4_K_M.gguf --kvcache-dtype turbo4

# Multi-shard 5.2 GGUF (auto-detected from HF subfolder)
xinfer --d 0,1,2,3 --m unsloth/Qwen3.5-122B-A10B-GGUF --f Q3_K_M --kvcache-dtype fp8

# FP8 on Metal
xinfer --m Qwen/Qwen3.5-4B-FP8

# Gemma4 26B (NVFP4)
xinfer --m unsloth/gemma-4-26b-a4b-it-NVFP4

# MLA model (GLM4.7 Flash)
xinfer --m GadflyII/GLM-4.7-Flash-NVFP4

# Interactive CLI chat
xinfer --i --m unsloth/Qwen3.5-27B-GGUF --f Qwen3.5-27B-Q4_K_M.gguf

# Faster GDN prefill on Hopper with slight precision loss
SM90_LOWER_PRECISION_GDN_PREFILL=1 xinfer --m Qwen/Qwen3.5-35B-A3B-FP8

# AWQ (pack-quantized)
xinfer --m cyankiwi/Qwen3-Coder-Next-AWQ-4bit

# MultiNode: GLM 5.2 (DeepSeek V3.2 architecture, FP8)
# Master node
xinfer --d 0,1,2,3,4,5,6,7 --m zai-org/GLM-5.2-FP8 --num-nodes 2 --node-rank 0 --master-addr 192.168.xxx.xxx
# Other node(s)
xinfer --d 0,1,2,3,4,5,6,7 --m zai-org/GLM-5.2-FP8 --num-nodes 2 --node-rank 1 --master-addr 192.168.xxx.xxx

# ISQ Q4K + FP8 KV cache
xinfer --m Qwen/Qwen3.6-35B-A3B --isq q4k --kvcache-dtype fp8

# ISQ Q4K + TurboQuant KV cache
xinfer --m Qwen/Qwen3.6-35B-A3B --isq q4k --kvcache-dtype turbo4

# Metal ISQ
xinfer --m /path/Qwen3-4B --isq q6k

# Multi-GPU GGUF (model ID + subfolder)
xinfer --d 0,1,2,3 --m unsloth/Qwen3.5-122B-A10B-GGUF --f Q3_K_M

# Remote GGUF — single-file model
xinfer --m unsloth/Qwen3.5-27B-GGUF --f Qwen3.5-27B-Q4_K_M.gguf

# Local GGUF file (direct path)
xinfer --d 0,1 --m /path/Qwen3-30B-A3B-Instruct-2507-Q4_K_M.gguf

# Local GGUF folder
xinfer --d 0,1,2,3 --m /path/to/Qwen3.5-122B-A10B-GGUF/Q3_K_M

# turbo4: 4-bit K+V — 3.7× compression, best tradeoff
xinfer --m Qwen/Qwen3.6-27B-FP8 --kvcache-dtype turbo4

# turbo3: 3-bit K + 4-bit V — 4.7× compression
xinfer --m Qwen/Qwen3.6-27B-FP8 --kvcache-dtype turbo3

# turbo8: FP8 K + 4-bit V — 2.6× compression, highest quality
xinfer --m Qwen/Qwen3.6-27B-FP8 --kvcache-dtype turbo8

# 35B MoE (NVFP4 + turbo4) — fits on single 24 GB GPU
xinfer --m unsloth/Qwen3.6-35B-A3B-NVFP4 --kvcache-dtype turbo4

# 30B MoE (GGUF Q4_K_M + turbo4) — consumer GPU
xinfer --m unsloth/Qwen3-30B-A3B-Instruct-2507-GGUF \
  --f Qwen3-30B-A3B-Instruct-2507-Q4_K_M.gguf --kvcache-dtype turbo4

# Upload images via built-in Chat UI or send image_url in API requests

# Qwen3.6 35B MoE (FP8, multimodal)
xinfer --m Qwen/Qwen3.6-35B-A3B-FP8 --ui-server

# Qwen3-VL 8B (GGUF)
xinfer --m unsloth/Qwen3-VL-8B-Instruct-GGUF --f Qwen3-VL-8B-Instruct-Q8_0.gguf --ui-server

# Gemma4 26B MoE (NVFP4, multimodal)
xinfer --m unsloth/gemma-4-26b-a4b-it-NVFP4 --ui-server

# Mistral-3 VL 3B (BF16, multimodal)
xinfer --m mistralai/Ministral-3-3B --ui-server

📘 Build from source code

Option 1 — Cargo

# Prerequisites: Rust compiler, CUDA Toolkit (optional) or Metal Xcode command line tool
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
sudo apt-get install -y git build-essential

export XINFER_REPO="https://github.com/guoqingbao/xinfer"
# MacOS/Metal: replace features to `metal`
# SM_70/SM_75 (e.g., V100): remove `flashinfer` and `cutlass` features
cargo install --git $XINFER_REPO xinfer --features cuda,nccl,flashinfer,cutlass

Option 2 — Docker

# Turing/V100 (sm_70/sm_75): remove `flashinfer` and `cutlass` features
./build_docker.sh "cuda,nccl,flashinfer,cutlass"

See Docker guide →

pip install maturin maturin[patchelf]

# FlashInfer backend (SM80+)
./build.sh --release --features cuda,nccl,flashinfer,cutlass,python

# Flash Attention backend
./build.sh --release --features cuda,nccl,flashattn,cutlass,python

# macOS Metal
maturin build --release --features metal,python

# Install
pip install target/wheels/xinfer*.whl --force-reinstall

See more Python examples →

🌐 Multi-Node Tensor Parallelism

Distribute inference across multiple machines using TCP-based NCCL bootstrap. No MPI required.

# Node 0 (master, 192.168.1.100): scheduler + API
xinfer --d 0,1,2,3 --m /data/DeepSeek-R1/ \
  --num-nodes 2 --node-rank 0 --master-addr 192.168.1.100 --ui-server

# Node 1 (worker, 192.168.1.101): forward-only daemon
xinfer --d 0,1,2,3 --m /data/DeepSeek-R1/ \
  --num-nodes 2 --node-rank 1 --master-addr 192.168.1.100

All nodes must have model weights locally and be TCP-reachable on --master-port (default 29500). See Get Started for details.

🔀 Prefill-Decode Disaggregation

Split prefill (prompt processing) and decode (token generation) across GPUs or machines. Eliminates decode stalls during long-context prefilling. PD Server and PD Client must use same KvCache type (--kvcache-dtype). API request(s) must send to PD Client and the PD Server only process internal prefill requests sent from PD Client.

Local IPC (multirank)

# PD Server (prefill GPU, default port 7000)
xinfer --d 0,1 --m Qwen/Qwen3-30B-A3B-Instruct-2507 --pd-server

# PD Client (decode GPU + API)
xinfer --d 2,3 --m /path/Qwen3-30B-A3B-Instruct-2507 --isq q4k --ui-server --port 8000 --pd-client

Multinode (tcp mode)

# Server machine (192.168.1.100)
target/release/xinfer --d 0,1 --m Qwen/... --pd-server --pd-url tcp://0.0.0.0:8100

# Client machine
target/release/xinfer --d 0,1 --m /path/... --pd-client --pd-url tcp://192.168.1.100:8100 --ui-server --port 8000

Metal/macOS requires --pd-url (no LocalIPC support).

mkdir -p /tmp/pd-sockets

# Server container
docker run --gpus '"device=0,1"' -v /tmp/pd-sockets:/sockets ...
target/release/xinfer --d 0,1 --m Qwen/... --pd-server --pd-url file:///sockets

# Client container
docker run --gpus '"device=2,3"' -v /tmp/pd-sockets:/sockets ...
target/release/xinfer --d 0,1 --m /path/... --pd-client --pd-url file:///sockets --ui-server --port 8000

🔌 MCP Tool Calling

xinfer --m unsloth/Qwen3-30B-A3B-Instruct-2507-GGUF \
  --f Qwen3-30B-A3B-Instruct-2507-Q4_K_M.gguf --ui-server --mcp-config ./mcp.json

MCP documentation →

🔌 Structured Outputs & Guided Decoding

Constraint-based generation via llguidance — JSON Schema, regex, Lark grammars, choice lists.

# Enable structured outputs and tool grammar
xinfer --m Qwen/Qwen3.6-27B-FP8 --ui-server --enable-tool-grammar

Structured outputs documentation →

📚 Documentation

Using Agents under xInfer backend: xbot · OpenCode · Kilo Code · Claude Code · Goose

⚙️ CLI Reference

Environment Variables

Example (Blackwell with high-precision NVFP4 decode):

XINFER_NVFP4_FORCE_LUT=1 xinfer --m nvidia/Qwen3-30B-A3B-FP4 --ui-server

Example (Hopper with faster GDN prefill):

SM90_LOWER_PRECISION_GDN_PREFILL=1 xinfer --m Qwen/Qwen3.5-35B-A3B-FP8 --ui-server

🛠️ Roadmap

• Batched inference (Metal)
• GGUF format support
• FlashAttention (CUDA)
• CUDA Graph
• OpenAI-compatible API (streaming support)
• Continuous batching
• Multi-gpu inference (Safetensors, GPTQ, AWQ, GGUF)
• Multi-node tensor parallelism (TCP-based NCCL bootstrap, no MPI required)
• Speedup prompt processing on Metal/macOS
• Chunked Prefill
• Prefix cache (available on CUDA when prefix-cache enabled)
• Model loading from hugginface hub
• Model loading from ModelScope (China)
• Prefix cache for Metal/macOS
• FP8 KV Cache (CUDA, all backends including FlashInfer on SM80+)
• FP8 KV Cache (Metal)
• FP8 KV Cache (with FlashInfer, SM80+)
• TurboQuant KV Cache (2-4 bit compression with WHT rotation)
• FP8 Models (CUDA: MoE, Dense; Metal: Dense)
• GLM 5.2 (DeepSeek V3.2 + DSA)
• Additional model support (Kimi K2, etc.)
• CPU KV Cache Offloading
• Prefill-decode Disaggregation (CUDA)
• Prefill-decode Disaggregation (Metal)
• Built-in ChatGPT-like Web Server
• Embedding API
• Tokenize/Detokenize API
• MCP Integration & Tool Calling
• Prefix Caching
• Claude/Anthropic-compatible API Server
• Support CUDA 13
• Support FlashInfer backend
• Support DeepGEMM backend (Hopper)
• MXFP4/NVFP4 Model Support
• Support Turboquant (4-bit, 3-bit) KvCache
• TentorRT-LLM
• Multi-token Prediciton (MTP)

📚 References

• Candle-vLLM
• Python nano-vllm

Star History

Like this project? Give it a ⭐ and contribute!