Supported Models

November 12, 2025 · View on GitHub

Supported Models

FastDeploy currently supports the following models, which can be downloaded automatically during FastDeploy deployment.Specify the model parameter as the model name in the table below to automatically download model weights (all supports resumable downloads). The following three download sources are supported:

When using automatic download, the default download source is AIStudio. Users can modify the default download source by setting the FD_MODEL_SOURCE environment variable, which can be set to “AISTUDIO”, ‘MODELSCOPE’ or “HUGGINGFACE”. The default download path is ~/ (i.e., the user's home directory). Users can modify the default download path by setting the FD_MODEL_CACHE environment variable, e.g.:

export FD_MODEL_SOURCE=AISTUDIO # "AISTUDIO", "MODELSCOPE" or "HUGGINGFACE"
export FD_MODEL_CACHE=/ssd1/download_models

Example launch Command using baidu/ERNIE-4.5-21B-A3B-PT:

python -m fastdeploy.entrypoints.openai.api_server \
       --model baidu/ERNIE-4.5-0.3B-PT \
       --port 8180 \
       --metrics-port 8181 \
       --engine-worker-queue-port 8182 \
       --max-model-len 32768 \
       --max-num-seqs 32

Large Language Models

These models accept text input.

Models	DataType	Example HF Model
⭐ERNIE	BF16\WINT4\WINT8\W4A8C8\WINT2\FP8	baidu/ERNIE-4.5-VL-424B-A47B-Paddle; baidu/ERNIE-4.5-300B-A47B-Paddle quick start best practice; baidu/ERNIE-4.5-300B-A47B-2Bits-Paddle; baidu/ERNIE-4.5-300B-A47B-W4A8C8-TP4-Paddle; baidu/ERNIE-4.5-300B-A47B-FP8-Paddle; baidu/ERNIE-4.5-300B-A47B-Base-Paddle; baidu/ERNIE-4.5-21B-A3B-Paddle; baidu/ERNIE-4.5-21B-A3B-Base-Paddle; baidu/ERNIE-4.5-21B-A3B-Thinking; baidu/ERNIE-4.5-VL-28B-A3B-Thinking; baidu/ERNIE-4.5-0.3B-Paddle quick start best practice; baidu/ERNIE-4.5-0.3B-Base-Paddle, etc.
⭐QWEN3-MOE	BF16/WINT4/WINT8/FP8	Qwen/Qwen3-235B-A22B; Qwen/Qwen3-30B-A3B, etc.
⭐QWEN3	BF16/WINT8/FP8	Qwen/qwen3-32B; Qwen/qwen3-14B; Qwen/qwen3-8B; Qwen/qwen3-4B; Qwen/qwen3-1.7B; Qwen/qwen3-0.6B, etc.
⭐QWEN2.5	BF16/WINT8/FP8	Qwen/qwen2.5-72B; Qwen/qwen2.5-32B; Qwen/qwen2.5-14B; Qwen/qwen2.5-7B; Qwen/qwen2.5-3B; Qwen/qwen2.5-1.5B; Qwen/qwen2.5-0.5B, etc.
⭐QWEN2	BF16/WINT8/FP8	Qwen/Qwen/qwen2-72B; Qwen/Qwen/qwen2-7B; Qwen/qwen2-1.5B; Qwen/qwen2-0.5B; Qwen/QwQ-32, etc.
⭐DEEPSEEK	BF16/WINT4	unsloth/DeepSeek-V3.1-BF16; unsloth/DeepSeek-V3-0324-BF16; unsloth/DeepSeek-R1-BF16, etc.
⭐GPT-OSS	BF16/WINT8	unsloth/gpt-oss-20b-BF16, etc.
⭐GLM-4.5/4.6	BF16/wfp8afp8	zai-org/GLM-4.5-Air; zai-org/GLM-4.6 最佳实践 etc.

Multimodal Language Models

These models accept multi-modal inputs (e.g., images and text).

Models	DataType	Example HF Model
ERNIE-VL	BF16/WINT4/WINT8	baidu/ERNIE-4.5-VL-424B-A47B-Paddle quick start best practice ; baidu/ERNIE-4.5-VL-28B-A3B-Paddle quick start best practice ; baidu/ERNIE-4.5-VL-28B-A3B-Thinking quick start best practice;
PaddleOCR-VL	BF16/WINT4/WINT8	PaddlePaddle/PaddleOCR-VL best practice ;
QWEN-VL	BF16/WINT4/FP8	Qwen/Qwen2.5-VL-72B-Instruct; Qwen/Qwen2.5-VL-32B-Instruct; Qwen/Qwen2.5-VL-7B-Instruct; Qwen/Qwen2.5-VL-3B-Instruct

More models are being supported. You can submit requests for new model support via Github Issues.