Deploy MiniCPM5-2B and MiniCPM5-1B with LiteRT-LM (Android / iOS / desktop / IoT)

September 10, 2026 · View on GitHub

LiteRT-LM is Google's on-device runtime for language models, built on LiteRT (formerly TensorFlow Lite). A model is packed once into a .litertlm bundle, and the same file runs on the CPU and the GPU of Android phones, iPhones, desktops and IoT boards such as the Raspberry Pi, through a command-line tool, an OpenAI-compatible local server, a Python package, and Kotlin / Swift APIs. Use this path when the target is a phone, an app, or a local server on a laptop.

The MiniCPM5-2B bundles are hosted in litert-community/MiniCPM5-2B; that card carries the conversion recipe and the measurements quoted here. The MiniCPM5-1B bundles are hosted in litert-community/MiniCPM5-1B. Every command on this page was run with litert-lm 0.17.0 on an Apple M4 Max; the Android check used litertlm-android 0.17.0 on a Galaxy S26.

TL;DR

uv tool install litert-lm

# first run downloads the 1.55 GB int4 bundle
litert-lm run --from-huggingface-repo=litert-community/MiniCPM5-2B MiniCPM5-2B_int4.litertlm \
    --prompt "What is the capital of France?"

The file lands in ~/.litert-lm/cache/huggingface/litert-community/MiniCPM5-2B/; later runs skip the download. GPU, the thinking switches, and sampling:

MODEL=~/.litert-lm/cache/huggingface/litert-community/MiniCPM5-2B/MiniCPM5-2B_int4.litertlm

litert-lm run "$MODEL" --backend gpu --prompt "1+1=?"
litert-lm run "$MODEL" --backend gpu --thinking false --prompt "1+1=?"
litert-lm run "$MODEL" --backend gpu --thinking-budget 2048 --prompt "1+1=?"
litert-lm run "$MODEL" --top-k 40 --top-p 0.95 --temperature 1.0 --thinking false --prompt "1+1=?"

Pre-converted bundles

Model	File	Recipe	Size	Tested on
MiniCPM5-2B	`MiniCPM5-2B_int4.litertlm`	int4 blockwise-32 + OCTAV on linears, int8 embedding	1.55 GB	CPU + GPU: Mac, Galaxy S26, iPhone 17 Pro
MiniCPM5-2B	`MiniCPM5-2B_int8.litertlm`	int8 dynamic linears + embedding, fp32 activations declared	2.60 GB	CPU + GPU: Mac, Galaxy S26 (iOS: see below)
MiniCPM5-1B	`minicpm_wi4b32_wi8_afp32.litertlm`	int4 block-32 linears, int8 embedding and LM head, fp32 activations	0.79 GB	CPU (Mac)
MiniCPM5-1B	`minicpm_wi4b32_wi8_afp32_gpu_opt.litertlm`	same recipe, optimized for GPU execution	0.79 GB	GPU (Mac)
MiniCPM5-1B	`MiniCPM5-1B_dynamic_wi8_afp32.litertlm`	int8 dynamic, fp32 activations	1.11 GB	CPU (Mac)

Which 2B file: int4 is the phone file (smaller, fastest GPU decode on both devices measured) and the right one for direct answers or short reasoning. int8 is the file when the reasoning has to complete: on the same questions its thinking chains are 3–4× shorter than int4's and close where int4's keep going. Both 2B files embed the checkpoint's own chat_template.jinja, so enable_thinking and the tool-calling format work unchanged, and both declare the thought channel (next section). The 2B repo also carries two CPU-only files described in its card.

The int8 file, here on the Mac's Metal GPU:

litert-lm run --from-huggingface-repo=litert-community/MiniCPM5-2B MiniCPM5-2B_int8.litertlm --backend gpu \
    --prompt "A train travels 60 km in 45 minutes. What is its average speed in km/h?"

MiniCPM5-1B:

litert-lm run --from-huggingface-repo=litert-community/MiniCPM5-1B minicpm_wi4b32_wi8_afp32.litertlm --prompt "1+1=?"
litert-lm run --from-huggingface-repo=litert-community/MiniCPM5-1B minicpm_wi4b32_wi8_afp32_gpu_opt.litertlm --backend gpu --prompt "1+1=?"

Thinking

Thinking is the default of the 2B files and the two 1B int4 files: with no thinking flag the model decides for itself and, in practice, reasons before every answer (MiniCPM5-1B_dynamic_wi8_afp32 is packaged with thinking off by default). The bundles declare the reasoning as a thought channel, so the CLI prints it between [thought] and [/thought] and the Kotlin / Swift APIs hand it to you separately (channels["thought"]); the answer text stays clean.

--thinking false: direct answers with no reasoning block; on 1+1=? (GPU) the turn took 3.0 s against 24.7 s with thinking on.
--thinking-budget N: caps the reasoning at N tokens. This model's chains run long, so start at 2048 and use 4096 for math.
Sampling: OpenBMB recommends temperature 1.0, top_p 0.95. The bundles ship a greedy sampler (temperature 0, no top-k), which stays greedy whatever the temperature, so pass all three: --top-k 40 --top-p 0.95 --temperature 1.0 (with --top-p --temperature alone the output stays byte-identical across seeds). The card's correctness numbers are greedy.
The KV cache takes the bundle's setting: 4096 tokens for the two 2B files and MiniCPM5-1B_dynamic_wi8_afp32, 1024 for the two 1B int4 files (--max-num-tokens to change it); the prompt format is ChatML, from the template embedded in each bundle.

OpenAI-compatible server

litert-lm serve exposes every bundle in the local registry on /v1/models and /v1/chat/completions (server guide). Import the file once under a name, start the server, then call it like any OpenAI endpoint:

litert-lm import --from-huggingface-repo=litert-community/MiniCPM5-2B MiniCPM5-2B_int4.litertlm minicpm5-2b
litert-lm serve --host 127.0.0.1 --port 9379

curl http://127.0.0.1:9379/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d '{"model": "minicpm5-2b", "messages": [{"role": "user", "content": "1+1=?"}], "reasoning_effort": "none"}'

"reasoning_effort": "none" gives a direct answer; any other value (low to xhigh) turns thinking on, and the reasoning is counted in usage.completion_tokens_details.reasoning_tokens. With thinking on, the reasoning needs room (the int4 file filled the 4096-token context on 1+1=? and returned no content); when the reasoning has to complete, use the int8 file. max_completion_tokens caps the reply, temperature, top_p, top_k and seed are honored, and "stream": true streams.

Android

AI Edge Gallery (no code). Install Google AI Edge Gallery, open Model manager, tap + and choose Import from HF, then paste the Hugging Face link of the .litertlm file (or download the file to the phone first and use From local model file). The litert-community card and the Gallery wiki have the details.

Kotlin API (your own app). The runtime is one dependency from Google Maven (Android guide):

// build.gradle.kts
implementation("com.google.ai.edge.litertlm:litertlm-android:0.17.0")

The GPU backend needs these entries in AndroidManifest.xml (Android guide); apps targeting Android 12 (API 31) or higher cannot load vendor libraries they do not declare, and without them initialize() succeeds and the first message fails with Can not find OpenCL library on this device:

<application>
    <uses-native-library android:name="libvndksupport.so" android:required="false" />
    <uses-native-library android:name="libOpenCL.so" android:required="false" />
</application>

import com.google.ai.edge.litertlm.*

val engine = Engine(EngineConfig(
    modelPath = modelFile.absolutePath,
    backend = Backend.GPU(),
    cacheDir = context.cacheDir.absolutePath,
))
engine.initialize()

engine.createConversation(ConversationConfig(maxOutputToken = 1024)).use { conversation ->
    val reply = conversation.sendMessage("Explain on-device AI in simple terms.")
    println(reply)
    println(reply.channels["thought"])
}
engine.close()

Tested on a Galaxy S26 (Snapdragon SM8850, Adreno) with litertlm-android 0.17.0 and the int4 file on Backend.GPU() (Backend.CPU() selects the CPU): the engine initializes with every node of every signature delegated to OpenCL, and the reply arrives with the reasoning in channels["thought"]. ConversationConfig also takes thinkingConfig = ThinkingConfig(enableThinking = false) or ThinkingConfig(enableThinking = true, thinkingTokenBudget = 2048) and a samplerConfig = SamplerConfig(topK = 40, topP = 0.95, temperature = 1.0), as in the Android guide. conversation.sendMessageAsync(...) streams.

iOS

Tested on an iPhone 17 Pro with the int4 file on both backends (Metal GPU and CPU; init 5.7 s and 2.2 s). The int8 file's main weight section is 2.33 GB, more than an iOS app maps in one piece with the default entitlements; the com.apple.developer.kernel.increased-memory-limit entitlement lifts that limit (single sections of 2.87 GB and 4.24 GB from other models have loaded and run on an iPhone 17 Pro with it), not checked with this file. The int4 file's main section is 1.28 GB. The Swift API is documented in the LiteRT-LM Swift guide (Swift Package Manager, https://github.com/google-ai-edge/LiteRT-LM); this page does not cover the Xcode steps.

Speed and accuracy

All numbers are from the litert-community card. Apple M4 Max, litert-lm benchmark 0.17.0, 256-token prefill / 256-token decode, 3 runs, 1024-token KV cache, no compiled-model cache (--cache no):

File	Backend	Prefill	Decode	Time to first token	Init
int4	GPU (Metal)	1699 tok/s	92.8 tok/s	0.16 s	3.7 s
int4	CPU	149 tok/s	31.1 tok/s	1.76 s	4.5 s
int8 (fp32 activations)	GPU (Metal)	1405 tok/s	74.7 tok/s	0.20 s	3.0 s
int8 (fp32 activations)	CPU	161 tok/s	30.0 tok/s	1.62 s	15.0 s

Galaxy S26 (Snapdragon SM8850, Adreno), the LiteRT-LM v0.16.0 release binary, 205-token prompt, 2 runs per cell (ranges shown; a reasoning model decodes its own full response, so decode lengths vary):

File	Backend	Prefill	Decode	Time to first token	Peak RSS
int4	GPU (OpenCL)	401–411 tok/s	16.1–18.6 tok/s	0.56 s	1.14 GB
int4	CPU	39–72 tok/s	15.6–15.8 tok/s	2.9–5.3 s	2.12 GB
int8 (fp32 activations)	GPU (OpenCL)	150–160 tok/s	10.9–12.8 tok/s	1.4 s	1.10 GB
int8 (fp32 activations)	CPU	103–157 tok/s	11.7 tok/s	1.4–2.1 s	2.90 GB

Accuracy, GSM8K first 100 test questions, greedy, thinking off, up to 2048 new tokens: bf16 PyTorch 92 %, int8 CPU 91 %, int4 CPU 86 %, int4 GPU 87 %. Both 2B files score 8/8 on the card's 8-question sanity gate on the Mac, CPU and GPU.

Common pitfalls

Thinking on the int4 file runs long chains: on the card's ten-question thinking-on subset (3584-token budget) it closes 0/10 on the CPU, where int8 closes 9/10 like the bf16 model. Pass --thinking false for direct answers; when the reasoning has to complete, use the int8 file.
int8 on iOS needs the increased-memory-limit entitlement. With the default entitlements the 2.33 GB weight section does not map; add com.apple.developer.kernel.increased-memory-limit (see iOS). int4's 1.28 GB main section is under the limit.
The output cap is what leaves no answer: when the reasoning runs into the context (--max-num-tokens, 4096 for the 2B files) or the app's maxOutputToken, the chain is cut and nothing follows. When the reasoning has to complete, keep those at 2048 or more (4096 for math); 1024, as in the snippets, is enough for short answers.
First run per backend is slow: the CPU run writes an XNNPACK weight cache beside the file and the GPU run compiles its kernels (--cache no skips the disk cache; the Kotlin cacheDir is the same mechanism).
GPU in your own Android app: without the <uses-native-library> entries above, initialize() succeeds and the first message fails with Can not find OpenCL library on this device.
Temperature without top-k does nothing: --temperature and --top-p only take effect together with --top-k above 1 (the default is greedy). The Kotlin SamplerConfig takes the same three fields.
Scripts: the CLI reads stdin; when it runs from a script or a job with an open pipe on stdin, redirect it (< /dev/null) or it waits for end-of-input before generating.