README.md

December 26, 2025 · View on GitHub

News

[2025/09] 🔥 RTP-LLM 0.2.0 release with enhanced performance and new features
[2025/01] 🚀 RTP-LLM now supports Prefill/Decode separation with detailed technical report
[2025/01] 🌟 Qwen series model and bert embedding model now supported on Yitian ARM CPU
[2024/06] 🔄 Major refactor: Scheduling and batching framework rewritten in C++, complete GPU memory management, and new Device backend
[2024/06] 🏗️ Multi-hardware support in development: AMD ROCm, Intel CPU and ARM CPU support coming soon

RTP-LLM is a Large Language Model (LLM) inference acceleration engine developed by Alibaba's Foundation Model Inference Team. It is widely used within Alibaba Group, supporting LLM service across multiple business units including Taobao, Tmall, Idlefish, Cainiao, Amap, Ele.me, AE, and Lazada.

RTP-LLM is a sub-project of the havenask project.

Key Features

🏢 Production Proven

Trusted and deployed across numerous LLM scenarios:

Taobao Wenwen
Alibaba's international AI platform, Aidge
OpenSearch LLM Smart Q&A Edition
Large Language Model based Long-tail Query Rewriting in Taobao Search

⚡ High Performance

Utilizes high-performance CUDA kernels, including PagedAttention, FlashAttention, FlashDecoding, etc.
Implements WeightOnly INT8 Quantization with automatic quantization at load time
Support WeightOnly INT4 Quantization with GPTQ and AWQ
Adaptive KVCache Quantization
Detailed optimization of dynamic batching overhead at the framework level
Specially optimized for the V100 GPU

🔧 Flexibility and Ease of Use

Seamless integration with the HuggingFace models, supporting multiple weight formats such as SafeTensors, Pytorch, and Megatron
Deploys multiple LoRA services with a single model instance
Handles multimodal inputs (combining images and text)
Enables multi-machine/multi-GPU tensor parallelism
Supports P-tuning models

🚀 Advanced Acceleration Techniques

Loads pruned irregular models
Contextual Prefix Cache for multi-turn dialogues
System Prompt Cache
Speculative Decoding

Getting Started

Benchmark and Performance

Learn more about RTP-LLM's performance in our benchmark reports:

Performance Benchmark Tool

Acknowledgments

Our project is mainly based on FasterTransformer, and on this basis, we have integrated some kernel implementations from TensorRT-LLM. We also draw inspiration from vllm, transformers, llava, and qwen-vl. We thank these projects for their inspiration and help.

Citation

If you find RTP-LLM useful in your research or project, please consider citing:

@Misc{rtp-llm,
  author       = {Alibaba},
  title        = {RTP-LLM: A High-Performance LLM Inference Engine},
  howpublished = {\url{https://github.com/alibaba/rtp-llm}},
  year         = {2025},
}