Dynamo Deployment with Aiconfigurator Guide

August 21, 2026 · View on GitHub

Note: This guide covers Dynamo platform deployments (--deployment-target dynamo-j2 or dynamo-python). For llm-d platform deployments, see the llm-d deployment examples in the README. The separate FPM V1 resource-workload workflow is summarized in Section 6.

This guide walks through

  • installing aiconfigurator
  • building the Dynamo container
  • generating configuration files (supports TRT-LLM, vLLM, and SGLang backends)
  • deploying Dynamo (single-node and two-node)
  • benchmarking the service and comparison

Take qwen3-32b-fp8 model as an example.

Note: This guide primarily uses TRT-LLM examples. For vLLM and SGLang examples with Dynamo, use the same workflow with --backend vllm or --backend sglang.

All-in-one Automation process

we're now supporting automate everything in one script, starting from configuring the deployment, generating the configs, preparing docker image and container, pulling model checkpoints, deploying the service, benchmarking and summarizing. Refer to Automation for more details.

Step-by-step Manual Deployment and Performance Alignment

Methodology

First, we need to define the problem we want to solve clearly. For a given model, we need to understand what's the constraint, we use ISL, OSL to define the target seqeunce length, and TTFT, TPOT to set the SLA constraint. Let's define the problem:

Can we find a config {parellel, concurrency} for given {ISL, OSL, Model, GPU}, which maximizes tokens/s/gpu, under TPOT and TTFT 
constraints.

Let's take a look at the pareto frontier,

  Pareto Frontier:
              Qwen/Qwen3-32B Pareto Frontier: tokens/s/gpu vs tokens/s/user          
    ┌──────────────────────────────────────────────────────────────────────────┐
2250┤ •• disagg                                                                │
    │ ff agg                                                                   │
    │ xx disagg best                                                           │
    │                                                                          │
1875┤  ff                                                                      │
    │   fff                                                                    │
    │     ff                                                                   │
    │      fff••                                                               │
1500┤         f •••                                                            │
    │         ff   ••••••••                                                    │
    │          ffff       •                                                    │
    │              f       •••••••                                             │
1125┤               ff            •                                            │
    │                ff            ••••                                        │
    │                  ffff            ••••x                                   │
    │                     fff              ••••                                │
 750┤                        fff               •                               │
    │                          ffffff           •                              │
    │                                ffffff      ••                            │
    │                                      fffffff ••••••                      │
 375┤                                             ff    •                      │
    │                                               fffffff•••••••••           │
    │                                                      ffffffffff          │
    │                                                                          │
   0┤                                                                          │
    └┬─────────────────┬──────────────────┬─────────────────┬─────────────────┬┘
     0                60                 120               180              240 
tokens/s/gpu                        tokens/s/user

Here the TPOT_limit=10ms. All datapoints shown on the pareto frontier meet TTFT_limit=1000ms
Each point on the pareto frontier can represent a different config {parallel, concurrency}.
The pareto frontier means, no matter how you change your deployment parallel strategy and benchmark with different concurrency, the datapoint will be under the frontier.
What we need is the highest point on the frontier which is left to 1000ms/TPOT_limit = 100 tokens/s/user. The point tagged as x is the one we find. This point indicates the parellel strategy as well as the concurrency level
We can find that, the config in this parallel strategy is potentially only best for this given concurrency instead of being generally better.
Thus we need corresponding benchmark way to make it work. Set concurrency sweep from 1 to target_concurrency predicted by aiconfigurator. E.g., [1 2 4 8 ... target_concurrency] Compare the result at target_concurrency with TTFT, TPOT, tokens/s/gpu and previous baseline you have

In order to reduce the impact of first batch of requests, we use concurrency * 10 as num_requests In order to avoid undefined cache hit rate when benchmarking with random data, we deliberately disable cache reuse to make it fair.

Step-by-step Manual Deployment

Problem:
16 H200 in total. QWen3 32B FP8.
ISL=4000, OSL=512, TTFT=300ms, TPOT=10ms, optimize tokens/s/gpu

If you would like to deploy by your own, when running the aiconfigurator cli exp|default, engine configuration files and executable scripts are automatically generated under the --save-dir, in the topx folder. The directory structure is:

results/Qwen_Qwen3-32B_h200_sxm_trtllm_isl4000_osl1000_ttft1000_tpot20_904495
├── agg
│   ├── best_config_topn.csv
│   ├── exp_config.yaml
│   ├── pareto.csv
│   ├── top1
│   │   ├── agg_config.yaml
│   │   ├── bench_run.sh          # aiperf benchmark sweep script (bare-metal)
│   │   ├── generator_config.yaml
│   │   ├── k8s_bench.yaml        # aiperf benchmark sweep Job (Kubernetes)
│   │   ├── k8s_deploy.yaml
│   │   └── run_0.sh
│   ...
├── disagg
│   ├── best_config_topn.csv
│   ├── exp_config.yaml
│   ├── pareto.csv
│   ├── top1
│   │   ├── bench_run.sh          # aiperf benchmark sweep script (bare-metal)
│   │   ├── decode_config.yaml
│   │   ├── generator_config.yaml
│   │   ├── k8s_bench.yaml        # aiperf benchmark sweep Job (Kubernetes)
│   │   ├── k8s_deploy.yaml
│   │   ├── prefill_config.yaml
│   │   ├── run_0.sh
│   │   └── run_1.sh  (for multi-node setups)
│   ...
└── pareto_frontier.png

Here, agg_config.yaml, prefill_config.yaml, and decode_config.yaml are TRTLLM engine configuration files, and run_x.sh are the executable scripts. k8s_deploy.yaml is for deployment in k8s. bench_run.sh and k8s_bench.yaml are benchmark helpers for running aiperf concurrency sweeps (see the CLI User Guide for details). In this guide, we're not using k8s.

For multi-node setups, there will be multiple run_x.sh scripts (one per node), each invoking the same TRTLLM engine config file. By default, run_0.sh starts both the frontend service and the workers, assuming ETCD and NATS are already running on node0, while other nodes only start the workers. Therefore, in multi-node deployments, set the head node IP via --generator-set ServiceConfig.head_node_ip=<IP> (there is no standalone --head_node_ip CLI flag).

Typically, the command is:

aiconfigurator cli default \
  --system h200_sxm \
  --model-path Qwen/Qwen3-32B \
  --isl 5000 \
  --osl 1000 \
  --ttft 2000 \
  --tpot 50 \
  --save-dir results \
  --total-gpus 16 \
  --generator-set ServiceConfig.model_path=/workspace/model_hub/Qwen3-32B-FP8 \
  --generator-set ServiceConfig.served_model_name=Qwen3-32B-FP8 \
  --generator-set ServiceConfig.head_node_ip=x.x.x.x

To customize parameters per worker type, override the Workers.<role> keys with --generator-set. To set worker counts, use WorkerConfig.* (e.g., WorkerConfig.prefill_workers=2). For example:

Run aiconfigurator cli default --generator-help to print information that is sourced directly from src/aiconfigurator/generator/config/deployment_config.yaml and backend_config_mapping.yaml.

aiconfigurator cli default \
  --system h200_sxm \
  --model-path Qwen/Qwen3-32B \
  --isl 5000 \
  --osl 1000 \
  --ttft 2000 \
  --tpot 50 \
  --save-dir results \
  --total-gpus 16 \
  --generator-set ServiceConfig.model_path=/workspace/model_hub/Qwen3-32B-FP8 \
  --generator-set ServiceConfig.served_model_name=Qwen3-32B-FP8 \
  --generator-set Workers.prefill.kv_cache_free_gpu_memory_fraction=0.8 \
  --generator-set ServiceConfig.head_node_ip=0.0.0.0

At runtime, copy the generated artifacts to each node, set up the engine configs directory, and execute the corresponding script:

# Create the engine_configs directory expected by the run scripts
mkdir -p /workspace/engine_configs

# Copy engine config files to the expected location (artifacts are directly under top1/, no nested agg/ or disagg/)
# For aggregated mode:
# cp ${your_save_dir}/.../agg/top1/agg_config.yaml /workspace/engine_configs/
# For disaggregated mode:
cp ${your_save_dir}/.../disagg/top1/*_config.yaml /workspace/engine_configs/

# Navigate to the generated top1 directory, then on node0:
cd ${your_save_dir}/.../disagg/top1
bash run_0.sh

# On other nodes
bash run_1.sh

Note: The generated configs are for deploying 1 replica instead of the cluster (defined as total_gpus). We'll bridge this gap in future.

Prerequisites

  • Docker with GPU support

1. Environment Setup

1.1 Install aiconfigurator

Use a minimal Ubuntu base image with python installed.

# Install Git LFS
apt-get update && apt-get install -y git-lfs

# Clone the repo
git clone https://github.com/ai-dynamo/aiconfigurator.git
cd aiconfigurator

# Install build tools and aiconfigurator
pip3 install "."

1.2 Build the Dynamo Container

In this example, we're using Dynamo 0.5.0, please switch to release/0.5.0 first.

# other version of trtllm can be used as well
# currently dynamo is at version 0.4.0, indicated in the tag
./container/build.sh \
  --framework TRTLLM \
  --tensorrtllm-pip-wheel tensorrt-llm==1.0.0rc6 \
  --tag dynamo:0.4.0-trtllm-1.0.0rc6

Please refer to Dynamo Getting Started for detailed dynamo installation

1.3 Download model checkpoint

HF_HUB_ENABLE_HF_TRANSFER=1 huggingface-cli download Qwen/Qwen3-32B-FP8 --local-dir /raid/hub/qwen3-32b-fp8

Please modify based on your own path '/raid/hub/qwen3-32b-fp8'

2. Running etcd and NATS

On Node 0, start etcd and NATS.io:

docker compose -f deploy/docker-compose.yml up -d

3. Single-Node Deployment

3.1 Generate Configuration with aiconfigurator

aiconfigurator cli default \
  --system h200_sxm \
  --isl 5000 \
  --osl 1000 \
  --ttft 1000 \
  --tpot 10 \
  --save-dir ./results \
  --model-path Qwen/Qwen3-32B \
  --total-gpus 8 \
  --generated-config-version 1.0.0rc4 \
  --generator-set ServiceConfig.head_node_ip=0.0.0.0 \
  --generator-set ServiceConfig.model_path=/workspace/model_hub/qwen3-32b-fp8 \
  --generator-set ServiceConfig.served_model_name=Qwen/Qwen3-32B-FP8 \
  --generator-set Workers.prefill.kv_cache_free_gpu_memory_fraction=0.9 \
  --generator-set Workers.decode.kv_cache_free_gpu_memory_fraction=0.5 \
  --generator-set Workers.agg.kv_cache_free_gpu_memory_fraction=0.7

We use 1.0.0rc3 (our latest data) for aiconfigurator and we can support generate configurations for running with trtllm 1.0.0rc4 worker.
--model-path is for aiconfigurator and --served_model_name is for dynamo deployment

For other supported configurations, please run aiconfigurator cli --help.

3.2 Verify Generated Configuration

Engine configuration files and executable scripts are automatically generated under the --save-dir. The directory structure is:

${save_dir}/
├── agg/
│   ├── top1/
│   │   ├── agg_config.yaml
│   │   ├── bench_run.sh          # aiperf benchmark sweep script (bare-metal)
│   │   ├── generator_config.yaml
│   │   ├── k8s_bench.yaml        # aiperf benchmark sweep Job (Kubernetes)
│   │   ├── k8s_deploy.yaml
│   │   └── run_0.sh
│   ├── best_config_topn.csv
│   ├── exp_config.yaml
│   └── pareto.csv
├── disagg/
│   ├── top1/
│   │   ├── bench_run.sh          # aiperf benchmark sweep script (bare-metal)
│   │   ├── decode_config.yaml
│   │   ├── generator_config.yaml
│   │   ├── k8s_bench.yaml        # aiperf benchmark sweep Job (Kubernetes)
│   │   ├── k8s_deploy.yaml
│   │   ├── prefill_config.yaml
│   │   ├── run_0.sh
│   │   └── run_1.sh  (for multi-node setups)
│   ├── best_config_topn.csv
│   ├── exp_config.yaml
│   └── pareto.csv
└── pareto_frontier.png

3.3 Launch the Dynamo Container

cd ..
docker run --gpus all --net=host --ipc=host \
  -v $(pwd):/workspace/mount_dir \
  -v /raid/hub:/workspace/model_hub/ \
  --rm -it dynamo:0.4.0-trtllm-1.0.0rc4

3.4 Deploy the service

Inside the container:

# Create the engine_configs directory expected by the run scripts
mkdir -p /workspace/engine_configs

# Copy engine config files to the expected location (artifacts are directly under top1/, no nested agg/ or disagg/)
# For disaggregated mode (recommended):
cp /workspace/mount_dir/${your_save_dir}/Qwen_Qwen3-32B_h200_sxm_trtllm_isl5000_osl1000_ttft1000_tpot10_*/disagg/top1/*_config.yaml /workspace/engine_configs/

# For aggregated mode:
# cp /workspace/mount_dir/${your_save_dir}/Qwen_Qwen3-32B_h200_sxm_trtllm_isl5000_osl1000_ttft1000_tpot10_*/agg/top1/agg_config.yaml /workspace/engine_configs/

# Navigate to the generated artifacts directory and launch dynamo
cd /workspace/mount_dir/${your_save_dir}/Qwen_Qwen3-32B_h200_sxm_trtllm_isl5000_osl1000_ttft1000_tpot10_*/disagg/top1
bash run_0.sh

Tip: If you see a Triton version mismatch error, reinstall Triton:

pip uninstall -y triton
pip install triton==3.3.1

3.5 Test the Service

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Accept: text/event-stream" \
  -d '{
    "model": "Qwen/Qwen3-32B-FP8",
    "messages": [
      { "role": "user", "content": "Introduce yourself" }
    ],
    "stream": true
  }'

3.6 Benchmark

4. Two-Node Deployment

4.1 Generate Configuration for Two Nodes

# ServiceConfig.head_node_ip (set via --generator-set below) must be the IP of node 0; etcd and NATS.io must already be running on node 0 (Step 2)
aiconfigurator cli default \
  --system h200_sxm \
  --isl 5000 \
  --osl 1000 \
  --ttft 200 \
  --tpot 8 \
  --save-dir ./ \
  --model-path Qwen/Qwen3-32B \
  --total-gpus 16 \
  --generator-set ServiceConfig.head_node_ip=NODE_0_IP \
  --generated-config-version 1.0.0rc4 \
  --generator-set ServiceConfig.model_path=/workspace/model_hub/qwen3-32b-fp8 \
  --generator-set ServiceConfig.served_model_name=Qwen/Qwen3-32B-FP8 \
  --generator-set Workers.prefill.kv_cache_free_gpu_memory_fraction=0.8 \
  --generator-set Workers.decode.kv_cache_free_gpu_memory_fraction=0.5 \
  --generator-set Workers.agg.kv_cache_free_gpu_memory_fraction=0.7

Note that even if --total-gpus 16, the optimal configuration generated by aiconfigurator may not require 16 GPUs. If only 8 GPUs are needed, it may produce just a run_0.sh, which can then be executed on each node.

Refer to the single node example to run the container on both node 0 and node 1.

4.2 Deploy on Node 0

Inside the container:

# Create the engine_configs directory expected by the run scripts
mkdir -p /workspace/engine_configs

# Copy engine config files to the expected location (artifacts are directly under top1/, no nested disagg/)
cp /workspace/mount_dir/Qwen_Qwen3-32B_h200_sxm_trtllm_isl5000_osl1000_ttft200_tpot8_*/disagg/top1/*_config.yaml /workspace/engine_configs/

# Navigate to the generated artifacts directory and launch dynamo on node 0 (includes frontend)
cd /workspace/mount_dir/Qwen_Qwen3-32B_h200_sxm_trtllm_isl5000_osl1000_ttft200_tpot8_*/disagg/top1
bash run_0.sh

4.3 Deploy on Node 1

Inside the container:

# Navigate to the same top1 directory and launch worker on node 1
cd /workspace/mount_dir/Qwen_Qwen3-32B_h200_sxm_trtllm_isl5000_osl1000_ttft200_tpot8_*/disagg/top1

# Launch dynamo on node 1 (workers only)
bash run_1.sh

4.4 Test the Service

curl http://NODE_0_IP:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Accept: text/event-stream" \
  -d '{
    "model": "Qwen/Qwen3-32B-FP8",
    "messages": [
      { "role": "user", "content": "Introduce yourself" }
    ],
    "stream": true
  }'

5. Deploy on Kubernetes

The generator can also emit a Kubernetes CR (k8s_deploy.yaml) for the K8S deployment.

For deploying Dynamo on Kubernetes, please refer to this dynamo/deploy and make sure to install the CRDs and platform first.

5.1 Generate Configuration for K8S

This produces disagg/k8s_deploy.yaml (and for Agg, agg/k8s_deploy.yaml) under --save-dir.

# Example (Disagg)
aiconfigurator cli default \
  --system h200_sxm \
  --isl 5000 \
  --osl 1000 \
  --ttft 200 \
  --tpot 8 \
  --save-dir ./ \
  --model-path Qwen/Qwen3-32B \
  --total-gpus 8 \
  --generated-config-version 1.0.0rc6 \
  --generator-set ServiceConfig.model_path=Qwen/Qwen3-32B-FP8 \
  --generator-set ServiceConfig.served_model_name=Qwen/Qwen3-32B-FP8 \
  --generator-set K8sConfig.k8s_image=nvcr.io/nvidia/ai-dynamo/tensorrtllm-runtime:0.7.0 \
  --generator-set K8sConfig.k8s_engine_mode=inline \
  --generator-set K8sConfig.k8s_model_cache=model-cache \
  --generator-set K8sConfig.k8s_namespace=dynamo-custom-ns \
  --generator-set Workers.prefill.kv_cache_free_gpu_memory_fraction=0.8 \
  --generator-set Workers.decode.kv_cache_free_gpu_memory_fraction=0.5 \
  --generator-set Workers.decode.cache_transceiver_backend=default

Since different TensorRT-LLM versions can require different config fields, set --generated-config-version to match the runtime used to deploy. For the TensorRT-LLM version corresponding to an official Dynamo image, refer to this pyproject, or check directly in the container via python -c "import tensorrt_llm; print(tensorrt_llm.__version__)". For nvcr.io/nvidia/ai-dynamo/tensorrtllm-runtime:0.5.0, use --generated-config-version 1.0.0rc6.

Apply (inline mode - default)

K8sConfig.k8s_engine_mode=inline

Inline mode embeds the engine configs into the Pod startup script; no ConfigMap is needed:

kubectl apply -f disagg/k8s_deploy.yaml
# or
kubectl apply -f agg/k8s_deploy.yaml

Additional arguments specific to Kubernetes

Use --generator-set K8sConfig.<field>=value (or place the same keys inside --generator-config). Defaults shown in bold.

  • K8sConfig.k8s_engine_mode={inline|configmap} - engine config delivery. inline by default.
  • K8sConfig.k8s_model_cache=<claimName> - optional model cache PVC mount (mounted at /workspace/model_cache). Leave it unset or empty to disable the mount. Specify the PVC name when you want pods to reuse an existing model cache; otherwise, if you directly set something like --generator-set ServiceConfig.model_path=Qwen/Qwen3-32B-FP8, the model is downloaded from Hugging Face and no PVC is required.
  • K8sConfig.k8s_hf_home=<path> - optional path for the HF_HOME environment variable in worker pods. When k8s_model_cache is configured but k8s_hf_home is not explicitly set, it automatically defaults to /workspace/model_cache to ensure HuggingFace libraries use the persistent volume. Set this to a custom path if you have a different volume mount structure.
  • K8sConfig.k8s_namespace=<ns> - target namespace. Default dynamo.
  • K8sConfig.k8s_image=<image> - runtime image. Default nvcr.io/nvidia/ai-dynamo/tensorrtllm-runtime:0.7.0.
  • K8sConfig.k8s_image_pull_secret=<secret> - optional pull secret name.

6. FPM V1 Resource Workload Workflow

--deployment-target fpm is a separate target for a vLLM single aggregated-worker deployment with exactly one worker replica. It emits a Pod for a single-node topology. A multinode topology emits a LeaderWorkerSet by default, or a Grove PodCliqueSet when K8sConfig.fpm_orchestrator: grove is selected. Router/planner configurations and invalid FPM topologies fail closed. It does not change the output of dynamo-j2, dynamo-python, or llm-d targets.

FPM V1 emits exactly three artifacts:

artifacts/
├── k8s_deploy.yaml   # keepalive Pod, LeaderWorkerSet, or PodCliqueSet
├── fpm_env.sh        # contract FPM_* exports: topology, rank/leader discovery, benchmark identity
└── run.sh            # sources fpm_env.sh, then launches the complete vLLM command in the foreground

The workload requests the generated image, per-node GPU limit, preserved custom resources, volumes, and mounts, but contains no engine arguments or engine/FPM environment variables. Add tokenized launch arguments through Workers.agg.extra_cli_args: list[str] and concrete {name, value} environment entries through K8sConfig.extra_env; the generator places both in run.sh. --benchmark-mode is required and accepts agg, prefill, or decode; it selects the runtime collection phase without changing the required single aggregated-worker topology. K8sConfig.fpm_shared_memory_size, K8sConfig.fpm_resource_labels, and K8sConfig.worker_extra_pod_spec.mainContainer.resources configure generated shared memory, workload/Pod labels, and non-GPU resource requests or limits. By default the FPM workload's main container renders Guaranteed QoS: CPU and memory are set with requests equal to limits, scaled per GPU (14 CPU cores and 64Gi of memory per GPU). Without such quota the Pod runs BestEffort for CPU, and launch-bound FPM measurements disperse across restarts with neighbour interference; requests==limits removes that spread. The defaults are skipped for any key (cpu or memory) already present in worker_extra_pod_spec.mainContainer.resources requests or limits, so explicit values always win. Grove output does not select a scheduler or queue by default; clusters that use KAI Scheduler can set worker_extra_pod_spec.schedulerName: kai-scheduler and fpm_resource_labels.kai.scheduler/queue explicitly. valueFrom, envFrom, and Secret-derived environment values are not supported in V1.

For a single-node Pod, an agent can create the resource once and execute the generated scripts in it:

kubectl apply -f artifacts/k8s_deploy.yaml
kubectl wait --for=condition=Ready pod/<pod> --timeout=10m
kubectl exec <pod> -- mkdir -p /tmp/fpm-bench
kubectl cp artifacts/fpm_env.sh <pod>:/tmp/fpm-bench/fpm_env.sh
kubectl cp artifacts/run.sh <pod>:/tmp/fpm-bench/run.sh
kubectl exec <pod> -- bash /tmp/fpm-bench/run.sh

This standalone flow demonstrates deployment only: run.sh just launches the engine, so completion gating and engine termination require the collector's staged runtime (fpm_exec.sh) and this sequence alone is not a complete measurement round.

For a multinode workload, the collector stages the complete runtime bundle on every Pod and starts the same scripts concurrently across them. fpm_env.sh derives rank and leader address from the selected controller's LWS or Grove environment (failing closed with exit 2), and run.sh appends the required model- or data-parallel coordination arguments. A multinode --dump-config-to path must contain {node_rank}; this substitution applies only to that option. Waiting for local result files and enforcing the schema-v2 completion contract belong to the collector's staged in-pod runtime, which consumes the FPM_* facts exported by fpm_env.sh. Strict plan/workload/repeat validation, result collection/aggregation/evidence, exit coordination, and cleanup remain collector responsibilities.

Each collection still starts a new engine and reloads the model. The collector's staged runtime stops result-producing engines after their local completion gate, refuses to overwrite existing benchmark outputs, and coordinates headless followers and final cleanup. By default /results is backed by Pod-local emptyDir, and those results disappear when the Pod is deleted; a matching user-provided volume and mount are preserved. Persistent engines and reuse of a GPU-resident model are outside the V1 scope. A target cluster needs the controller selected by K8sConfig.fpm_orchestrator; any scheduler and queue required by that cluster must be supplied through the generic Pod-spec and label overlays. Multinode GB200 configurations that enable MNNVL also include the generated ComputeDomain in k8s_deploy.yaml; single-node output never emits one. See the CLI User Guide for the full input example.

The current vLLM template matrix tops out at 0.20.1; reference 0.24.0-only flags may be passed through, but their runtime compatibility is not yet validated by the generator.