TensorRT-LLM User Guide

July 7, 2026 · View on GitHub

What is TensorRT-LLM

TensorRT-LLM (TRT-LLM) is an open-source library designed to accelerate and optimize the inference performance of large language models (LLMs) on NVIDIA GPUs. Built on PyTorch, TRT-LLM offers an easy-to-use Python LLM API that lets you serve any HuggingFace model directly, incorporating state-of-the-art optimizations to ensure efficient inference on NVIDIA GPUs.

How to run TRT-LLM models with Triton Server via the TensorRT-LLM backend

The TensorRT-LLM Backend lets you serve TensorRT-LLM models with Triton Inference Server. With the PyTorch backend (LLM API) you can serve any HuggingFace model directly — no engine compilation required. The steps below get you from an empty container to a running server in a few minutes.

Launch the container

docker run --rm -it --net host --shm-size=2g --ulimit memlock=-1 --gpus all \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    nvcr.io/nvidia/tritonserver:25.12-trtllm-python-py3 bash

Replace 25.12 with the latest tag from NGC.

Clone TRT-LLM and set your model

git clone https://github.com/NVIDIA/TensorRT-LLM.git

Edit TensorRT-LLM/triton_backend/all_models/llmapi/tensorrt_llm/1/model.yaml and set model: to any HuggingFace model ID or local path, for example:

model: TinyLlama/TinyLlama-1.1B-Chat-v1.0

All keys in model.yaml map directly to LLM() constructor arguments. This is where you configure KV cache, quantization, parallelism, and more. For gated models (e.g. Llama), set your token first: export HF_TOKEN=hf_...

Launch and test

Run the launch script from the parent of TensorRT-LLM/ (running it from inside the cloned folder causes ModuleNotFoundError: No module named 'tensorrt_llm.bindings'):

python3 TensorRT-LLM/triton_backend/scripts/launch_triton_server.py \
    --model_repo=TensorRT-LLM/triton_backend/all_models/llmapi/

Once the server is up, send a request:

curl -X POST localhost:8000/v2/models/tensorrt_llm/generate \
    -d '{"text_input": "The future of AI is", "sampling_param_max_tokens": 50}' | jq

For multi-GPU, multi-node, and the full set of configuration and deployment options, see the TensorRT-LLM Backend README and the LLM API guide.

Advanced Configuration Options and Deployment Strategies

Explore advanced configuration options and deployment strategies to optimize and run Triton with your TRT-LLM models effectively:

  • Model Deployment: Techniques for efficiently deploying and managing your models in various environments.
  • Multi-Instance GPU (MIG) Support: Run Triton and TRT-LLM models with MIG to optimize GPU resource management.
  • Scheduling: Configure scheduling policies to control how requests are managed and executed.
  • Key-Value Cache: Utilize KV cache and KV cache reuse to optimize memory usage and improve performance.
  • Decoding: Advanced methods for generating text, including top-k, top-p, top-k top-p, beam search, Medusa, and speculative decoding.
  • Chunked Context: Splitting the context into several chunks and batching them during generation phase to increase overall throughput.
  • Quantization: Apply quantization techniques to reduce model size and enhance inference speed.
  • LoRa (Low-Rank Adaptation): Use LoRa for efficient model fine-tuning and adaptation.

Tutorials

Make sure to check out the tutorials repo to see more guides on serving popular LLM models with Triton Server and TensorRT-LLM, as well as deploying them on Kubernetes.

Benchmark

GenAI-Perf is a command line tool for measuring the throughput and latency of LLMs served by Triton Inference Server. Check out the Quick Start to learn how to use GenAI-Perf to benchmark your LLM models.

Performance Best Practices

Check out the Performance tuning guide to learn how to optimize your TensorRT-LLM models for better performance.

Metrics

Triton Server provides metrics indicating GPU and request statistics. See the Triton Metrics section in the TensorRT-LLM Backend repo to learn how to query the Triton metrics endpoint to obtain TRT-LLM statistics.

Ask questions or report issues

Can't find what you're looking for, or have a question or issue? Feel free to ask questions or report issues in the GitHub issues page: