Deploy MiniCPM5-2B with vLLM on Ascend NPU (vllm-ascend)

vllm-ascend is the official Ascend NPU backend for vLLM. It ships as a platform plugin: you install it alongside a matching vLLM release, and vLLM auto-discovers it at import time — the model code, the OpenAI-compatible server and the sampling API are all identical to the CUDA path described in vllm.md.

MiniCPM5-2B uses the LlamaForCausalLM architecture, so it runs on the generic Ascend path with no custom kernels and no --model-impl override.

Install

The path of least resistance is the prebuilt image, which already contains CANN, torch_npu and both wheels:

docker run -it --net=host --shm-size=1g \
    --device /dev/davinci0 --device /dev/davinci_manager \
    --device /dev/devmm_svm --device /dev/hisi_hdc \
    -v /usr/local/Ascend/driver:/usr/local/Ascend/driver \
    -v /usr/local/dcmi:/usr/local/dcmi \
    -v /usr/local/bin/npu-smi:/usr/local/bin/npu-smi \
    quay.io/ascend/vllm-ascend:v0.18.0

To install into an existing CANN environment instead:

pip install vllm==0.18.0 vllm-ascend==0.18.0 \
    --extra-index-url https://download.pytorch.org/whl/cpu

torch resolves to the CPU build (2.9.0+cpu) — that is correct and expected. Device support comes from torch_npu, not from torch itself.

Environment setup

Source the CANN environment before every run. Without it the plugin loads but device init fails. Adjust these paths if CANN or NNAL is installed elsewhere:

source /usr/local/Ascend/ascend-toolkit/set_env.sh
source /usr/local/Ascend/nnal/atb/set_env.sh

The vllm_ascend_C extension links against the torch and torch_npu shared libraries, so they must be on the loader path:

export LD_LIBRARY_PATH=$(python3 -c "import torch,os;print(os.path.dirname(torch.__file__))")/lib:\
$(python3 -c "import torch_npu,os;print(os.path.dirname(torch_npu.__file__))")/lib:$LD_LIBRARY_PATH

Select devices with ASCEND_RT_VISIBLE_DEVICES — CUDA_VISIBLE_DEVICES has no effect:

export ASCEND_RT_VISIBLE_DEVICES=0        # single card
# export ASCEND_RT_VISIBLE_DEVICES=0,1,2,3  # four cards, pair with --tensor-parallel-size 4

Verify the toolchain before loading a model:

npu-smi info                      # health, HBM usage, running processes
python3 -c "import torch, torch_npu; print(torch_npu.npu.is_available(), torch_npu.npu.device_count())"
# True 8

If npu-smi fails with libc_sec.so: cannot open shared object file, the driver libraries are missing from the loader path. Add them: export LD_LIBRARY_PATH=/usr/local/Ascend/driver/lib64:/usr/local/Ascend/driver/lib64/common:/usr/local/Ascend/driver/lib64/driver:$LD_LIBRARY_PATH

OpenAI-compatible server

Identical to the CUDA invocation — only the environment differs:

export ASCEND_RT_VISIBLE_DEVICES=0
vllm serve /path/to/MiniCPM5-2B \
    --served-model-name MiniCPM5-2B \
    --dtype bfloat16 \
    --max-model-len 32768 \
    --gpu-memory-utilization 0.85 \
    --tensor-parallel-size 1 \
    --trust-remote-code \
    --port 8000

Confirm the plugin took over — this line must appear in the startup log:

INFO [__init__.py:239] Platform plugin ascend is activated

Tuning knobs

Flag	Default	When to change
--max-model-len	131072 (native 128K)	drop to 8192 / 32768 to cut KV-cache reservation and shorten warmup
--gpu-memory-utilization	0.9	fraction of HBM, not host RAM; lower it when sharing a card
--tensor-parallel-size	1	must equal the number of devices in ASCEND_RT_VISIBLE_DEVICES
--dtype	bfloat16	keep bf16; 910B has native bf16 support and fp16 buys nothing
--enforce-eager	unset	skips ACL graph capture — use to isolate a graph-mode bug, at a real latency cost

Ascend-specific notes

• --quantization / AWQ / GPTQ are not interchangeable with the CUDA kernels. Ascend quantization goes through msModelSlim; a GPTQ checkpoint built for CUDA will not load.
• Graph capture logs say "CUDA graphs". The progress bar reads Capturing CUDA graphs (mixed prefill-decode, PIECEWISE) even on NPU — the string is hardcoded upstream in vLLM. The captured graphs are ACL graphs. Cosmetic only.
• First start is slow. Warmup (profile + KV cache alloc + graph capture) takes ~75 s for a 2B model on one 910B3. torch.compile artifacts are cached under ~/.cache/vllm/torch_compile_cache/, so subsequent starts are faster.
• EngineCore died unexpectedly at exit is normal. For offline (LLM(...)) scripts this is teardown noise printed after your results, not a failure. Check whether your output was produced before treating it as an error.

The Docker command above is a minimal single-device example. Device nodes and driver mounts may differ across Ascend hardware and image versions; for multi-card serving, expose the required devices and set --tensor-parallel-size accordingly.

Chat completions

curl http://localhost:8000/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d '{
        "model": "MiniCPM5-2B",
        "messages": [{"role": "user", "content": "用一句话解释什么是张量并行。"}],
        "temperature": 1.0,
        "top_p": 0.95,
        "max_tokens": 1024,
        "chat_template_kwargs": {"enable_thinking": true}
    }'

MiniCPM5-2B is a think-only model. Use enable_thinking: true, temperature: 1.0, and top_p: 0.95 for generation.

Thinking is on by default. The bundled chat template enables it when enable_thinking is absent, and the reasoning text is returned inline inside a <think>...</think> block in message.content (not in a separate reasoning field, which stays null unless you configure a reasoning parser). MiniCPM5-2B is think-only, so keep enable_thinking enabled. Budget max_tokens accordingly — a short limit can be consumed by the <think> block before the final answer starts.

Sample run

$ curl -sS http://localhost:8000/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d '{"model":"MiniCPM5-2B","messages":[{"role":"user","content":"1+1=?"}],
         "temperature":1.0,"top_p":0.95,"max_tokens":256,
         "chat_template_kwargs":{"enable_thinking":true}}'
{
  "choices": [{
    "message": {"role": "assistant", "content": "<think>...</think>2"},
    "finish_reason": "stop"
  }]
}

Streaming ("stream": true) and GET /v1/models are available through the standard vLLM OpenAI-compatible server.

Offline / batched inference

from vllm import LLM, SamplingParams

llm = LLM(
    model="/path/to/MiniCPM5-2B",
    dtype="bfloat16",
    max_model_len=4096,
    gpu_memory_utilization=0.85,
    tensor_parallel_size=1,
    trust_remote_code=True,
)

out = llm.chat(
    [[{"role": "user", "content": "用一句话解释 GQA。"}]],
    SamplingParams(temperature=1.0, top_p=0.95, max_tokens=512),
    chat_template_kwargs={"enable_thinking": True},
)
print(out[0].outputs[0].text)

Run it with the environment from Environment setup sourced:

source /usr/local/Ascend/ascend-toolkit/set_env.sh
source /usr/local/Ascend/nnal/atb/set_env.sh
export ASCEND_RT_VISIBLE_DEVICES=0
python3 offline_infer.py

Tool calling

MiniCPM5-2B emits XML-style tool calls:

<function name="get_weather"><param name="city">Beijing</param></function>

The tags <function, <param, </param> and </function> are special tokens in this tokenizer — set "skip_special_tokens": false if you want to see them in raw /v1/completions output.

No generic parser handles this shape: the other XML-style parsers wrap calls in <tool_call> and write parameters as <parameter=city>, not <param name="city">. MiniCPM5 needs its own parser, which vLLM has shipped as minicpm5 since v0.23.0 (v0.22.1 and earlier do not have it).

Parsing happens in vLLM's OpenAI server layer, above the platform backend, so none of this is Ascend-specific: vllm-ascend is not involved and the behaviour matches CUDA exactly.

Requirements

vllm-ascend pins the same version number as vLLM ("vLLM (the same version as vllm-ascend)", upstream README), so the parser comes with the 0.23.0 stack as a whole:

Component	Version
vllm-ascend / vLLM	0.23.0
CANN	9.1.0
torch / torch-npu	2.10.0 / 2.10.0.post4

Serve

No plugin and no patch — the parser is built in:

vllm serve /path/to/MiniCPM5-2B \
    --served-model-name MiniCPM5-2B \
    --dtype bfloat16 --max-model-len 32768 --port 8000 \
    --enable-auto-tool-choice \
    --tool-call-parser minicpm5

Passing tools in a request implies tool_choice: "auto", which requires both flags. Without them the request fails with:

"auto" tool choice requires --enable-auto-tool-choice and --tool-call-parser to be set

Request

curl http://localhost:8000/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d '{
        "model": "MiniCPM5-2B",
        "messages": [{"role": "user", "content": "What is the weather in Beijing?"}],
        "tools": [{
            "type": "function",
            "function": {
                "name": "get_weather",
                "description": "Get current weather for a city",
                "parameters": {
                    "type": "object",
                    "properties": {"city": {"type": "string"}},
                    "required": ["city"]
                }
            }
        }],
        "tool_choice": "auto",
        "temperature": 1.0, "top_p": 0.95, "max_tokens": 512,
        "chat_template_kwargs": {"enable_thinking": true}
    }'

Non-streaming and streaming agree:

// tool needed
"content": "",
"tool_calls": [{
  "id": "call_91c904e4d81aa931",
  "type": "function",
  "function": {"name": "get_weather", "arguments": "{\"city\": \"Beijing\"}"}
}],
"finish_reason": "tool_calls"

// no tool needed — no spurious call
"content": "Hello.",
"tool_calls": [],
"finish_reason": "stop"

These responses were produced with the same upstream parser file, run on an older vLLM with a compatibility shim; the 0.23.0 stack itself was not available on the test hardware.

References

• vllm-ascend · docs
• MiniCPM vLLM (CUDA) deployment guide
• Ascend CANN