TensorRT Edge-LLM
July 2, 2026 · View on GitHub
TensorRT Edge-LLM
High-Performance Large Language Model Inference Framework for NVIDIA Edge Platforms
Overview | Quick Start | Performance | Documentation | Roadmap
Overview
TensorRT Edge-LLM is NVIDIA's high-performance C++ inference runtime for Large Language Models (LLMs) and Vision-Language Models (VLMs) on embedded platforms. It enables efficient deployment of state-of-the-art language models on resource-constrained devices such as NVIDIA Jetson, NVIDIA DRIVE, and NVIDIA DGX Spark platforms. TensorRT Edge-LLM provides convenient Python scripts to convert HuggingFace checkpoints to ONNX. Engine build and end-to-end inference runs entirely on Edge platforms.
Getting Started
For the supported platforms, models and precisions, see the Overview. Get started with TensorRT Edge-LLM in <15 minutes. For complete installation and usage instructions, see the Quick Start Guide.
Documentation
Introduction
- Overview - What is TensorRT Edge-LLM and key features
- Supported Models - Complete model compatibility matrix
- Checkpoint Exporter - Recommended ONNX export pipeline
User Guide
- Installation - Set up quantization,
tensorrt_edgellm, and the C++ runtime - Quick Start Guide - Run your first inference in ~15 minutes
- Examples - End-to-end workflows
- Quantization - Create quantized checkpoints for
tensorrt_edgellm - Experimental High-Level Python API and Server - vLLM-style API and OpenAI-compatible server
- Input Format Guide - Request format and specifications
- Chat Template Format - Chat template configuration
Developer Guide
Software Design
- Quantization Package Design - Quantization package architecture
- Engine Builder - Building TensorRT engines
- C++ Runtime Overview - Runtime system architecture
Advanced Topics
- Customization Guide - Customizing TensorRT Edge-LLM for your needs
- TensorRT Plugins - Custom plugin development
- Tests - Comprehensive test suite for contributors
Performance
See the Performance Benchmarks page for released benchmark results covering LLM and VLM prefill, generation throughput, memory usage, and EAGLE speculative decoding speedups.
Use Cases
🚗 Automotive
- In-vehicle AI assistants
- Voice-controlled interfaces
- Scene understanding
- Driver assistance systems
🤖 Robotics
- Natural language interaction
- Task planning and reasoning
- Visual question answering
- Human-robot collaboration
🏭 Industrial IoT
- Equipment monitoring with NLP
- Automated inspection
- Predictive maintenance
- Voice-controlled machinery
📱 Edge Devices
- On-device chatbots
- Offline language processing
- Privacy-preserving AI
- Low-latency inference
Featured Websites
- TensorRT Edge-LLM Jetson AI Lab tutorial
- Maximizing Memory Efficiency to Run Bigger Models on NVIDIA Jetson
- Build Next-Gen Physical AI with Edge-First LLMs for Autonomous Vehicles and Robotics
- Accelerate AI Inference for Edge and Robotics with NVIDIA Jetson T4000 and NVIDIA JetPack 7.1
- Accelerating LLM and VLM Inference for Automotive and Robotics with NVIDIA TensorRT Edge-LLM
Follow our GitHub repository for the latest updates, releases, and announcements.
Support
- Documentation: Full Documentation
- Quick Start: Quick Start Guide
- Roadmap: Developer Roadmap
- Issues: GitHub Issues
- Discussions: GitHub Discussions
- Forums: NVIDIA Developer Forums
License
Contributing
We welcome contributions! Please see our Contributing Guidelines for details.