full-api-reference.mdx

Kind: type Appears in: OperatorConfiguration

Kind: type Appears in: CheckpointStorageConfiguration

Kind: type Appears in: CheckpointStorageConfiguration

Kind: type Appears in: CheckpointStorageConfiguration

Kind: type Appears in: CheckpointConfiguration

Kind: type Appears in: CheckpointConfiguration

NOTE: auto-detection here only verifies that the resource.k8s.io/v1 API is registered on the apiserver (Kubernetes 1.34+). It does NOT verify that a GPU-specific DRA resource driver (e.g. nvidia/k8s-dra-driver-gpu) is installed, that its DeviceClass exists, or that node-level GPU drivers are compatible. An admin can use enabled: false to force-off DRA integration on clusters where the API is present but the GPU driver stack is not wired up — this makes the operator fail GMS / inter-pod failover admissions early with a clear error instead of letting pods Pend with a confusing "resourceclaim not found" at schedule time.

Kind: type Appears in: OperatorConfiguration

Kind: enum Underlying type: string Appears in: DiscoveryConfiguration

Allowed values

• kubernetes DiscoveryBackendKubernetes is the Kubernetes discovery backend
• etcd DiscoveryBackendEtcd is the etcd discovery backend

Operator Default Values Injection

The Dynamo operator automatically applies default values to various fields when they are not explicitly specified in your deployments. These defaults include:

• Health Probes: Startup, liveness, and readiness probes are configured differently for frontend, worker, and planner components. For example, worker components receive a startup probe with a 2-hour timeout (720 failures × 10 seconds) to accommodate long model loading times.

• Security Context: All components receive fsGroup: 1000 by default to ensure proper file permissions for mounted volumes. This can be overridden via the extraPodSpec.securityContext field.

• Shared Memory: All components receive an 8Gi shared memory volume mounted at /dev/shm by default (can be disabled or resized via the sharedMemory field).

• Environment Variables: Components automatically receive environment variables like DYN_NAMESPACE, DYN_PARENT_DGD_K8S_NAME, DYNAMO_PORT, and backend-specific variables.

• Pod Configuration: Default terminationGracePeriodSeconds of 60 seconds and restartPolicy: Always.

• Autoscaling: When enabled without explicit metrics, defaults to CPU-based autoscaling with 80% target utilization.

• Backend-Specific Behavior: For multinode deployments, probes are automatically modified or removed for worker nodes depending on the backend framework (VLLM, SGLang, or TensorRT-LLM).

Pod Specification Defaults

All components receive the following pod-level defaults unless overridden:

• terminationGracePeriodSeconds: 60 seconds
• restartPolicy: Always

Security Context

The operator automatically applies default security context settings to all components to ensure proper file permissions, particularly for mounted volumes:

• fsGroup: 1000 - Sets the group ownership of mounted volumes and any files created in those volumes

This default ensures that non-root containers can write to mounted volumes (like model caches or persistent storage) without permission issues. The fsGroup setting is particularly important for:

• Model downloads and caching
• Compilation cache directories
• Persistent volume claims (PVCs)
• SSH key generation in multinode deployments

Overriding Security Context

To override the default security context, specify your own securityContext in the extraPodSpec of your component:

services:
  YourWorker:
    extraPodSpec:
      securityContext:
        fsGroup: 2000  # Custom group ID
        runAsUser: 1000
        runAsGroup: 1000
        runAsNonRoot: true

Important: When you provide any securityContext object in extraPodSpec, the operator will not inject any defaults. This gives you complete control over the security context, including the ability to run as root (by omitting runAsNonRoot or setting it to false).

OpenShift and Security Context Constraints

In OpenShift environments with Security Context Constraints (SCCs), you may need to omit explicit UID/GID values to allow OpenShift's admission controllers to assign them dynamically:

services:
  YourWorker:
    extraPodSpec:
      securityContext:
        # Omit fsGroup to let OpenShift assign it based on SCC
        # OpenShift will inject the appropriate UID range

Alternatively, if you want to keep the default fsGroup: 1000 behavior and are certain your cluster allows it, you don't need to specify anything - the operator defaults will work.

Shared Memory Configuration

Shared memory is enabled by default for all components:

• Enabled: true (unless explicitly disabled via sharedMemory.disabled)
• Size: 8Gi
• Mount Path: /dev/shm
• Volume Type: emptyDir with memory medium

To disable shared memory or customize the size, use the sharedMemory field in your component specification.

Health Probes by Component Type

The operator applies different default health probes based on the component type.

Frontend Components

Frontend components receive the following probe configurations:

Liveness Probe:

• Type: HTTP GET
• Path: /health
• Port: http (8000)
• Initial Delay: 60 seconds
• Period: 60 seconds
• Timeout: 30 seconds
• Failure Threshold: 10

Readiness Probe:

• Type: Exec command
• Command: curl -s http://localhost:${DYNAMO_PORT}/health | jq -e ".status == \"healthy\""
• Initial Delay: 60 seconds
• Period: 60 seconds
• Timeout: 30 seconds
• Failure Threshold: 10

Worker Components

Worker components receive the following probe configurations:

Liveness Probe:

• Type: HTTP GET
• Path: /live
• Port: system (9090)
• Period: 5 seconds
• Timeout: 30 seconds
• Failure Threshold: 1

Readiness Probe:

• Type: HTTP GET
• Path: /health
• Port: system (9090)
• Period: 10 seconds
• Timeout: 30 seconds
• Failure Threshold: 60

Startup Probe:

• Type: HTTP GET
• Path: /live
• Port: `system$ (9090)
• \text{Period}: 10 \text{seconds}
• \text{Timeout}: 5 \text{seconds}
• \text{Failure} \text{Threshold}: 720 (\text{allows} \text{up} \text{to} 2 \text{hours} \text{for} \text{startup}: 10\text{s} \times 720 = 7200\text{s})

[!\text{NOTE}] \text{For} \text{larger} \text{models} (\text{typically} >70\text{B} \text{parameters}) \text{or} \text{slower} \text{storage} \text{systems}, \text{you} \text{may} \text{need} \text{to} \text{increase} \text{the} $failureThresholdto allow more time for model loading. Calculate the required threshold based on your expected startup time:failureThreshold = (expected_startup_seconds / period)`. Override the startup probe in your component specification if the default 2-hour window is insufficient.

Multinode Deployment Probe Modifications

For multinode deployments, the operator modifies probes based on the backend framework and node role:

VLLM Backend

The operator automatically applies distributed execution configuration based on parallelism settings:

Tensor/Pipeline Parallel Mode (Recommended) (when world_size > GPUs_per_node):

• Uses PyTorch multiprocessing (mp) backend for distributed execution (--distributed-executor-backend mp)
• Supports multi-node deployments with PyTorch's native distributed initialization
• All nodes: Run vLLM with proper --nnodes, --node-rank, --master-addr flags injected
• Probes: Worker probes adjusted; leader probes remain active

Ray Backend:

• Used for use cases such as Elastic EP
• Install with pip install "ray>=2.55.0" and configure --distributed-executor-backend ray

Data Parallel Mode (when world_size × data_parallel_size > GPUs_per_node):

• Worker nodes: All probes (liveness, readiness, startup) are removed
• Leader nodes: All probes remain active

SGLang Backend

• Worker nodes: All probes (liveness, readiness, startup) are removed

TensorRT-LLM Backend

• Leader nodes: All probes remain unchanged
• Worker nodes:
  • Liveness and startup probes are removed
  • Readiness probe is replaced with a TCP socket check on SSH port (2222):
    • Initial Delay: 20 seconds
    • Period: 20 seconds
    • Timeout: 5 seconds
    • Failure Threshold: 10

Environment Variables

The operator automatically injects environment variables into component containers based on component type, backend framework, and operator configuration. User-provided envs values always take precedence over operator defaults.

All Components

These environment variables are injected into every component container regardless of type.

Infrastructure (Conditional)

These are injected into all components when the corresponding infrastructure service is configured in the operator's OperatorConfiguration.

Frontend Components

Worker Components

Planner Components

EPP (Endpoint Picker Plugin) Components

VLLM Backend

TensorRT-LLM Backend

Service Accounts

The following component types automatically receive dedicated service accounts:

• Planner: planner-serviceaccount
• EPP: epp-serviceaccount

Image Pull Secrets

The operator automatically discovers and injects image pull secrets for container images. When a component specifies a container image, the operator:

1. Scans all Kubernetes secrets of type kubernetes.io/dockerconfigjson in the component's namespace
2. Extracts the docker registry server URLs from each secret's authentication configuration
3. Matches the container image's registry host against the discovered registry URLs
4. Automatically injects matching secrets as imagePullSecrets in the pod specification

This eliminates the need to manually specify image pull secrets for each component. The operator maintains an internal index of docker secrets and their associated registries, refreshing this index periodically.

To disable automatic image pull secret discovery for a specific component, add the following annotation:

annotations:
  nvidia.com/disable-image-pull-secret-discovery: "true"

Autoscaling Defaults

When autoscaling is enabled but no metrics are specified, the operator applies:

• Default Metric: CPU utilization
• Target Average Utilization: 80%

Port Configurations

Default container ports are configured based on component type:

Frontend Components

• Port: 8000
• Protocol: TCP
• Name: http

Worker Components

• Port: 9090 (system)
• Protocol: TCP
• Name: system
• Port: 19090 (NIXL)
• Protocol: TCP
• Name: nixl

Planner Components

• Port: 9085
• Protocol: TCP
• Name: metrics

EPP Components

• Port: 9002 (gRPC)
• Protocol: TCP
• Name: grpc
• Port: 9003 (gRPC health)
• Protocol: TCP
• Name: grpc-health
• Port: 9090 (metrics)
• Protocol: TCP
• Name: metrics

Backend-Specific Configurations

VLLM

• Ray Head Port: 6379 (for Ray-based multinode deployments)
• MP Master Port: 29500 (for PyTorch distributed multinode TP/PP deployments with mp backend)
• Data Parallel RPC Port: 13445 (for data parallel multinode deployments)

SGLang

• Distribution Init Port: 29500 (for multinode deployments)

TensorRT-LLM

• SSH Port: 2222 (for multinode MPI communication)
• OpenMPI Environment: OMPI_MCA_orte_keep_fqdn_hostnames=1

Implementation Reference

For users who want to understand the implementation details or contribute to the operator, the default values described in this document are set in the following source files:

• Health Probes, Security Context & Pod Specifications: internal/dynamo/graph.go - Contains the main logic for applying default probes, security context, environment variables, shared memory, and pod configurations
• Component-Specific Defaults:
  • internal/dynamo/component_common.go - Base container and pod spec shared by all component types
  • internal/dynamo/component_frontend.go
  • internal/dynamo/component_worker.go
  • internal/dynamo/component_planner.go
  • internal/dynamo/component_epp.go
• Image Pull Secrets: internal/secrets/docker.go - Implements the docker secret indexer and automatic discovery
• Backend-Specific Behavior:
  • internal/dynamo/backend_vllm.go
  • internal/dynamo/backend_sglang.go
  • internal/dynamo/backend_trtllm.go
• Checkpoint / Restore:
  • internal/checkpoint/podspec.go - Checkpoint env var injection and volume setup
  • internal/checkpoint/resolve.go - Checkpoint resolution logic
  • internal/checkpoint/resource.go - Checkpoint resource management
• Constants & Annotations: internal/consts/consts.go - Defines annotation keys and other constants

Notes

• All these defaults can be overridden by explicitly specifying values in your DynamoComponentDeployment or DynamoGraphDeployment resources
• User-specified probes (via livenessProbe, readinessProbe, or startupProbe fields) take precedence over operator defaults
• For security context, if you provide any securityContext in extraPodSpec, no defaults will be injected, giving you full control
• For multinode deployments, some defaults are modified or removed as described above to accommodate distributed execution patterns
• The extraPodSpec.mainContainer field can be used to override probe configurations set by the operator

Raw generated Markdown source: docs/fern/pages/reference/kubernetes-api/additional-resources/api-reference-k8s.md