Labeler Configuration

September 22, 2026 · View on GitHub

Overview

The Labeler module automatically applies labels to Kubernetes nodes based on GPU runtime components. It watches DCGM and driver pods deployed by GPU Operator and detects Kata Containers runtime. This document covers all Helm configuration options for system administrators.

Labels Applied

The labeler automatically manages these node labels:

Label	Values	Purpose
`nvsentinel.dgxc.nvidia.com/dcgm.version`	`3.x`, `4.x`	DCGM major version detected from DCGM pods
`nvsentinel.dgxc.nvidia.com/driver.installed`	`true`, `false`	NVIDIA driver pod status on node
`nvsentinel.dgxc.nvidia.com/kata.enabled`	`true`, `false`	Kata Containers runtime presence
`nvsentinel.dgxc.nvidia.com/gpu.count.current`	non-negative integer	Current GPU count from the configured class expression
`nvsentinel.dgxc.nvidia.com/gpu.count.expected`	non-negative integer	Expected GPU count from override or learned hardware-class baseline
`nvsentinel.dgxc.nvidia.com/nic.count.current`	non-negative integer	Current NIC count from the configured class expression
`nvsentinel.dgxc.nvidia.com/nic.count.expected`	non-negative integer	Expected NIC count from override or learned hardware-class baseline

Configuration Reference

Module Enable/Disable

Controls whether the labeler module is deployed in the cluster.

global:
  labeler:
    enabled: true

Resources

Defines CPU and memory resource requests and limits for the labeler pod.

labeler:
  resources:
    requests:
      cpu: 100m
      memory: 128Mi
    limits:
      cpu: 500m
      memory: 256Mi

Logging

Sets the verbosity level for labeler logs.

labeler:
  logLevel: info  # Options: debug, info, warn, error

Kubernetes API Rate Limits

The labeler inherits the Kubernetes client limits from global.qps and global.burst (defaults: 5 and 10). Set component values only when the labeler needs different limits:

labeler:
  qps: 40
  burst: 80

Positive qps values enable client-side throttling, 0 uses the client-go default, and a negative value disables client-side throttling. burst must be non-negative; 0 uses the client-go default.

Pre-Installed Drivers

Assumes NVIDIA drivers are installed directly on the host rather than via GPU Operator driver containers. When enabled, the labeler sets nvsentinel.dgxc.nvidia.com/driver.installed=true on all GPU nodes it manages (nvidia.com/gpu.present=true; nodes opted out with nvsentinel.dgxc.nvidia.com/managed=false — for example during external remediation — are excluded), skipping driver pod detection.

labeler:
  assumeDriverInstalled: false

DCGM Pod Discovery

Comma-separated values of the app pod label used to find the DCGM pod on each node. The default matches both the classic GPU Operator DCGM DaemonSet and the GPUCluster (DRA) mode one, so the correct pod is found in either mode without per-cluster configuration.

labeler:
  dcgmAppLabel: "nvidia-dcgm,nvidia-dcgm-dra"

Driver Pod Discovery

Pod label values used to find the driver pod on each node: the app label of the GPU Operator driver pod, and the k8s-app label of the GKE driver installer pod. The defaults match GPU Operator and GKE.

labeler:
  driverAppLabel: "nvidia-driver-daemonset"
  gkeInstallerAppLabel: "nvidia-driver-installer"

DCGM Bootstrap Gating

Controls whether the DCGM pod must be ready before the DCGM version label is set for the first time on a node.

labeler:
  requireDCGMReadyForBootstrap: true

Kata Containers Detection

Configures detection of Kata Containers runtime on nodes.

labeler:
  kataLabelOverride: ""

Parameters

kataLabelOverride

Optional custom node label to check for Kata Containers detection, in addition to the default label.

Default Label: katacontainers.io/kata-runtime

When empty, only the default label is checked. When set, both default and custom labels are checked.

Truthy Values

The following label values (case-insensitive) are considered truthy for Kata detection:

"true"
"enabled"
"1"
"yes"

Any other value or missing label results in kata.enabled=false.

Expected Device Counts

Expected device-count labeling is disabled by default. When enabled, the labeler evaluates enabled classes and writes current/expected count labels only when the configured CEL expression returns a valid non-negative integer.

The Helm chart renders this values block into a TOML ConfigMap entry and mounts it into the labeler pod. Because expressions are compiled at startup, Helm also annotates the pod template with a checksum so changes to the ConfigMap roll the Deployment.

labeler:
  expectedDeviceCounts:
    enabled: true
    classes:
      - name: gpu
        enabled: true
        labels:
          current: nvsentinel.dgxc.nvidia.com/gpu.count.current
          expected: nvsentinel.dgxc.nvidia.com/gpu.count.expected
        groupingLabels:
          - node.kubernetes.io/instance-type
          - nvidia.com/gpu.product
        expectedCountOverrides:
          - matchLabels:
              nvidia.com/gpu.product: NVIDIA-GB200
            count: 8
        currentExpression: |
          int(node.metadata.labels['nvidia.com/gpu.count'])

The CEL context exposes:

node: the cached projection of the Kubernetes Node being reconciled.
resourceSlices: ResourceSlice objects associated with the node.
sum(list<int>): helper that returns the sum of a list of integers.

Node fields available to expressions

To limit informer memory use, the Labeler does not cache complete Node objects. The following fields are always retained:

metadata.name, metadata.uid, and metadata.resourceVersion
all metadata.labels
the nvsentinel.dgxc.nvidia.com/dcgm-bootstrap-completed annotation, when present

When expected device counts have at least one enabled class, the Labeler also retains status.allocatable and status.capacity. Device-count expressions that read Node data must use node.metadata.labels, node.status.allocatable, or node.status.capacity.

All other Node fields are discarded before caching, including spec, other annotations, status.conditions, addresses, images, and node information. When expected device counts are disabled, all of status is discarded. Expressions that reference discarded fields are unsupported and receive only the field's empty or absent value.

For classes without a matching override, the expected value is learned as the maximum current or existing expected count among nodes with the same configured grouping-label values. Learned expected counts can rise automatically, but do not fall automatically when a node reports fewer devices.

Kata Detection Examples

Example 1: Default Detection

labeler:
  kataLabelOverride: ""

Checks only katacontainers.io/kata-runtime label on nodes.

Example 2: Custom Kata Label

labeler:
  kataLabelOverride: "io.katacontainers.config.runtime.oci_runtime"

Checks both katacontainers.io/kata-runtime and io.katacontainers.config.runtime.oci_runtime. Kata is enabled if either label has a truthy value.

GPU Operator Integration

The labeler watches for specific pod labels to detect DCGM and driver status.

Expected Pod Labels

DCGM Pods:

metadata:
  labels:
    app: nvidia-dcgm

Driver Pods:

metadata:
  labels:
    app: nvidia-driver-daemonset

If your GPU Operator configures its operands with different labels, the labeler will not detect the components.