tt-inference-server

tt-inference-server is the fastest way to deploy and test models for serving inference on Tenstorrent hardware.

Quickstart guide

On first-run please see the prerequisites guide for general Tenstorrent hardware and software setup.

For the specific quickstart guide and details for your model, select your model and hardware configuration in Model Support pages and tables below. Alternatively you can see all models supported for your given Tenstorrent hardware.

Models by Model Type

Browse models by type:

• LLM Models - Large Language Models
• VLM Models - Vision-Language Models
• Video Models - Video generation models
• Image Models - Image generation models
• Audio Models - Speech-to-text models
• TTS Models - Text-to-speech models
• Embedding Models - Text embedding models
• CNN Models - Convolutional Neural Networks

Models by Hardware Configuration

Browse models by hardware:

• Dual WH Galaxy
• Quad WH Galaxy
• WH Galaxy
• BH LoudBox
• BH 4xP150
• BH P300
• BH QuietBox 2
• p100
• p150
• WH LoudBox/QuietBox
• n150
• n300

Workflow automation in tt-inference-server

For details on the workflow automation for:

• deploying inference servers
• running E2E performance benchmarks
• running accuracy evals

See:

• Model Readiness Workflows User Guide.
• workflows/README.md

Benchmarking

For more details see benchmarking/README.md

Evals

For more details see evals/README.md

Development

Developer documentation: docs/README.md

Release documentation: scripts/release/README.md

If you encounter setup or stability problems with any model please file an issue and our team will address it.