v26.10.0.md

September 9, 2026 · View on GitHub

This is the GA release of the DOCA Platform Framework (DPF). It includes bug fixes and improvements to enhance the provisioning and orchestration of NVIDIA BlueField DPUs in Kubernetes environments.

Revision History

DateDescription
August 2026General Availability (GA) release of DOCA Platform Framework v26.10.0

Features

  • Per-DPU BMC credentials for Zero Trust provisioning

    • Description: DPF now supports per-device BMC credentials for DPUDevice resources in Zero Trust deployments. Each DPU can use its own unique BMC password by referencing a Kubernetes Secret in the DPUDevice.spec.bmcCredentialSecretName field. When set, the per-device password takes precedence over the shared bmc-shared-password secret. The feature also supports password rotation by changing the referenced secret name, and reports credential health through a new BMCCredentialsReady status condition.
    • For details, see Per-DPU BMC Credentials.

  • BMC factory reset and password hardening

    • Description: While it initializes a DPUDevice, DPF now resets that DPU's BMC to factory defaults once, before configuring anything on it, so every DPU starts from a known state. Immediately afterwards it applies the password from the BMC credential secret to every account it manages, which on BlueField-4 includes the ssh-only service account in addition to the Redfish user, so no managed account is left on the BMC factory default password. Progress is reported through a new BMCFactoryResetReady condition and status.bmcFactoryResetRequestTime, which together guarantee the BMC is reset at most once per DPUDevice.
    • Opting out: A factory reset erases all BMC configuration, including its network settings, so it is safe only for BMCs that get their address over DHCP. Set DPUDevice.spec.bmcFactoryResetPolicy: Never for a statically addressed BMC, or DPFOperatorConfig.spec.provisioningController.installInterface.installViaRedfish.discoveredDPUDeviceBMCFactoryResetPolicy: Never to apply that to every DPUDevice created by discovery.
    • For details, see BMC Factory Reset.

  • DPU node labeling via scripts

    • Description: DPF now supports applying Kubernetes Node labels to DPU cluster nodes during provisioning via executable scripts. The DPU agent scans a configurable directory (/var/lib/dpf/dpuagent/node-label-scripts/ by default) for executable scripts, runs each one, and applies its output as Node labels after kubelet startup. Each non-empty stdout line must have the form <label-key-suffix>=<label-value>, so a single script can report multiple labels; keys are namespaced under the scripts.dpu.nvidia.com/ prefix. Scripts can be delivered to the DPU via DPUFlavor.spec.configFiles. This enables operators to attach hardware-discovered or environment-specific labels to DPU nodes automatically during provisioning.
    • For details, see DPU Node Label Scripts.

  • Automatic DPU BMC mTLS server certificate rotation

    • Description: In Zero Trust deployments the provisioning controller now connects to each DPU BMC over Redfish with full mTLS certificate verification, and automatically rotates the BMC mTLS server certificate before it expires. The certificate is issued for 365 days and is renewed within a configurable window (default 30 days before expiry) via DPFOperatorConfig.spec.provisioningController.bmcServerCertRenewBefore. Each DPUDevice reports its certificate expiry and rotation state under status.bmcServerCertificate and through a new BMCServerCertificateReady status condition. For DPUs provisioned before this release, the controller backfills the expiry from the certificate the BMC is currently serving instead of forcing an immediate fleet-wide rotation.
    • Manual rotation: An individual DPU's BMC server certificate can be rotated on demand by annotating the DPUDevice with provisioning.dpu.nvidia.com/rotate-bmc-server-certificate (any non-empty, changing value, e.g. a timestamp).
    • Expired certificate recovery: If a BMC server certificate is already expired, automatic and manual rotation cannot start because the provisioning controller cannot open the verified mTLS connection to the BMC. Recovery requires deleting and recreating the affected DPUDevice so DPF re-runs mTLS bootstrap and installs a fresh BMC server certificate.
    • For details, see BMC mTLS Server Certificate Rotation.

  • Self-signed provisioning CA rotation

    • Description: DPF can now rotate the self-signed provisioning CA without disrupting trust for bfb-registry, the provisioning controller, DPU BMC mTLS, HostAgent, or the DPU OS. Rotation uses a dual-trust window: DPF adds the new CA to the dpf-ca-trust-bundle ConfigMap alongside the old CA, consumers converge on the combined bundle, the provisioning issuer switches to the new CA, and DPF removes the old CA only after the new certificate chain has been applied everywhere. Updates to dpf-ca-trust-bundle are propagated automatically.

      In Zero Trust deployments, each DPUDevice syncs the bundle to the BMC truststore and reports progress through status.caTrustBundle and the CATrustBundleReady condition. On the DPU OS, dpu-agent applies the bundle and records status.agentStatus.trustBundleHash and trustBundleLastUpdateTime.

    • For details, see Provisioning CA Certificate Rotation.

  • Outdated status field on DPU

    • Description: DPU resources now expose a status.outdated field that is set by the DPUSet controller whenever the parent DPUSet template has drifted in a way that requires the DPU to be reprovisioned. The field is presence-based: it appears only while the DPU is out of date and is removed automatically once the DPU is back in sync with the template.
    • Shape: 
      status:
  outdated:
    timeStamp: "2026-07-15T08:30:00Z"
    reason: OutdatedBFB
    message: "DPU template has changed (BFB: bfb-v1 -> bfb-v2)."
    • When it appears: one of the following template fields has changed relative to the running DPU: bfb, dpuFlavor, secureBoot, or bluefieldSoftware. The field is emitted regardless of the DPUSet strategy.type (OnDelete or RollingUpdate). Non-disruptive template changes (e.g. nodeEffect, cluster node labels/annotations) do not produce this field.
    • Reasons: OutdatedBFB, OutdatedDPUFlavor, OutdatedSecureBoot, OutdatedBlueFieldSoftware. When multiple fields have diverged, reason reports the first in fixed precedence order (BFB -> DPUFlavor -> SecureBoot -> BlueFieldSoftware) and message lists every detected diff.
    • TimeStamp: timeStamp records when the current drift was first observed. It is preserved across reconciles as long as reason is unchanged, and is refreshed when reason transitions.
    • Required action when present:
      • Under RollingUpdate, no user action is required — the DPUSet controller reprovisions affected DPUs automatically; status.outdated clears on each DPU once it matches the template again.
      • Under OnDelete, the operator must delete the affected DPU CR to trigger reprovisioning. The DPUSet controller will recreate it using the current template, and status.outdated will disappear on the new DPU once it matches.
      # Inspect outdated DPUs across the namespace (works under either strategy).
kubectl -n $DPF_NAMESPACE get dpu \
  -o jsonpath='{range .items[?(@.status.outdated)]}{.metadata.name}{"\t"}{.status.outdated.reason}{"\t"}{.status.outdated.message}{"\n"}{end}'

# Reprovision an outdated DPU (OnDelete strategy only).
kubectl -n $DPF_NAMESPACE delete dpu $DPU_NAME

  • Privileged pod enforcement for DPUService workloads

    • Description: DPF now restricts which DPUService workloads may run privileged containers in a DPUCluster.

      By default, a workload managed by a DPUService that targets a DPUCluster (spec.deployInCluster: false) may not run a container with securityContext.privileged: true unless the DPUService opts in via the new DPUService.spec.security.privileged field.

      Enforcement is implemented with a ValidatingAdmissionPolicy installed into each DPUCluster; opt-in can also be set on a DPUServiceTemplate.spec.security.privileged, which the DPUDeployment controller propagates to generated DPUServices.

      For DPUCluster-targeted DPUServices, spec.security.privileged must be set explicitly.

      This feature only gates securityContext.privileged: true; other privilege-escalation vectors (hostPID, hostIPC, hostNetwork, capabilities, hostPath, etc.) are not.

    • Default in this release: when spec.security.privileged is left unset, DPF currently defaults to allowing privileged workloads, so workloads carried over from earlier releases keep running. This default is planned to change to deny in a future release. Set spec.security.privileged explicitly now to avoid a behavior change on a later upgrade.

    • For details, see Privileged Pod Enforcement.

  • Global toggle for privileged pod enforcement

    • Description: Privileged pod enforcement can now be disabled at runtime via the new DPFOperatorConfig.spec.security.privilegedPodEnforcement field (defaults to true).

      Setting it to false takes effect promptly without restarting any component: the enforcement policy is switched to Audit mode instead of being deleted, so privileged pods are logged to the Kubernetes audit log rather than denied.

      Enforcement is fail-secure — it is treated as enabled when the field is unset All Security-related configuration on the DPFOperatorConfig is grouped under spec.security (for example spec.security.kata).

    • For details, see Privileged Pod Enforcement.

  • Reduced Prometheus metric cardinality across DPF scrapers

    • Description: All DPF-managed scrape configurations now apply metric allowlists that keep only the metrics consumed by the DPF dashboards and alert/recording rules. This covers the management cluster control plane, the DPF controllers, kube-state-metrics, and the DPU cluster control planes scraped via the kamaji-generated ServiceMonitors. In a reference deployment with one DPUCluster this reduces the total Prometheus TSDB from 245,438 to 100,481 active series (a ~59% cut). Per scrape job, before and after:

      LayerJobBeforeAfter
      Managementapiserver (built-in)123,06047,764
      Managementkube-state-metrics9,2454,525
      Managementdoca-platform-framework (DPF controllers)3,0403,040
      Managementkube-controller-manager / kube-scheduler4,457 / 4,9171,636 / 74
      Kamaji tenantapiserver / kcm / scheduler (per DPUCluster)58,748 / 4,359 / 3,46911,818 / 1,741 / 36
      DPU clusterkube-state-metrics per DPUCluster2,133960

    • Scalability: The last two rows scale with the number of DPUClusters. Each additional DPUCluster previously added roughly 68,700 series to the management Prometheus; it now adds roughly 14,600, about 4.7x less. This lowers Prometheus memory and storage growth per DPUCluster accordingly and allows a single management cluster to monitor significantly larger DPU fleets with the same Prometheus footprint.

      Within a DPUCluster, series grow with the number of DPUs and DPUServices, and the allowlists bound those growth rates as well: pod state metrics are capped at ~13 series per pod (instead of ~60 unfiltered), so a DPUService deployed as a DaemonSet adds ~13 series per DPU it runs on, and each DPU node adds ~73 kube_node_* series. The DPF resource state metrics on the management cluster (dpu_*, dpuservice_*, servicechain_*, ...) add a handful of series per custom resource.

      Projected series added to the management Prometheus for a single DPUCluster, assuming DPUServices are deployed as DaemonSets (one pod per DPU). The estimates combine the constant tenant control plane cost with the measured per-DPU, per-pod and per-resource costs; before-values use the unfiltered per-unit costs:

      DPUsDPUServicesBefore (est.)After (est.)
      15~67,000~13,900
      1010~74,500~16,600
      10020~203,000~55,200
      1,00020~1,420,000~427,000

  • DPU cluster workload metrics streaming and DPUService performance dashboard

    • Description: The OpenTelemetry Collector running on each DPU cluster now scrapes container, pod, and node resource metrics from every DPU node's kubelet and streams them to the management cluster, where they are written into Prometheus and tagged with the originating DPUCluster name (the cluster label). This makes DPUService CPU and memory consumption on the DPU cluster visible from the management cluster's Prometheus and Grafana, alongside the DPU control-plane metrics that were already collected. Metrics streaming is enabled by setting DPFOperatorConfig.spec.monitoring.openTelemetryCollector.metrics.endpoint to the management cluster collector endpoint, mirroring the existing logging.endpoint. The bundled kube-prometheus-stack values enable the Prometheus remote-write receiver so the collector can push these metrics.
    • Dashboard: A new Grafana dashboard, DOCA Platform Framework - DPUService Performance, presents per-namespace and top-pod CPU and memory working-set usage across DPU clusters, filterable by DPU cluster and namespace.
    • For details, see DPF-Operator-Managed Components and Grafana Dashboards.

  • DPUFlavorTemplate for per-DPU flavor rendering

    • Description: DPF now supports rendering a per-DPU DPUFlavor from a shared DPUFlavorTemplate body using values from DPUDevice.spec.values. Operators can express device-specific configuration (addresses, tags, hostnames, per-port settings, …) with a single template instead of maintaining one static DPUFlavor per DPU. Both DPUSet (spec.dpuTemplate.spec.dpuFlavorTemplate) and DPUDeployment (spec.dpus.flavorTemplate) accept a template reference, mutually exclusive with the static DPUFlavor reference. Each template-mode DPU carries labels that track the source template and the render input hashes. When an input changes, the DPU is marked outdated and re-rendered according to the update strategy: automatically under RollingUpdate, or after the operator deletes the DPU CR under OnDelete.
    • For details, see DPUFlavorTemplate: Per-DPU Flavor Rendering.

  • BFB Registry now served over HTTPS

    • Description: The bfb-registry is now served exclusively over an HTTPS endpoint, secured with a server certificate issued from the DPF CA. All clients that communicate with the bfb-registry now use HTTPS with TLS verification, including the HostAgent (BFB download), the DPUAgent (NIC firmware download), the DPU BMC (BFB install over Redfish), and the DPU OS dpu-agent apt repository in zero-trust deployments.

  • TLS verification for the customer-supplied container registry proxy

    • Description: When DPUFlavor.spec.containerdConfig.registryEndpoint is configured, DPF now enables TLS certificate verification for the DPU containerd registry mirror for nvcr.io instead of skipping it. The registry proxy server certificate is validated against the DPF CA trust bundle, which the DPU OS trusts via the dpf-ca-trust-bundle ConfigMap (data key ca.crt) delivered through cloud-init. To use a customer-supplied registry proxy, add the proxy's CA certificate to the dpf-ca-trust-bundle ConfigMap.

  • Standard CNI plugins now shipped by DPF

    • Description: DPF now bundles and installs the standard containernetworking/plugins CNI binaries (host-device, loopback, dhcp, static, vrf) alongside the DPF-owned rdma and ovs CNIs via the cni-installer DaemonSet. Previously these standard plugins were provided by the kubernetes-cni package baked into the BFB, which was outdated and absent when the BFB was built on a non-Ubuntu distribution. By shipping the plugins from DPF, /opt/cni/bin on the DPU is now fully owned by DPF regardless of what the BFB provides.

  • Opt-in NoExecute taint for host network readiness

    • Description: DPF can now apply a NoExecute taint to a host worker node when its Ready-phase DPU reports HostNetworkReady != True (for example, after the DPU reboots or host VFs disappear), evicting workloads that depend on host networking instead of merely blocking new scheduling. The feature is opt-in and disabled by default; enable it by setting DPFOperatorConfig.spec.dpuServiceController.disableHostNetworkReadyNoExecuteTaints to false. It is independent from, and can be combined with, the existing critical DPUService NoSchedule taint. DPF system hostNetwork DaemonSets already tolerate the new taint; customer-managed hostNetwork workloads that must stay on the node need to add the matching toleration.
    • For details, see Host Network Readiness NoExecute Taints.

Fixed Issues from Previous Release

  • HBN NAT offload was not applied on physical uplinks
    • HBN-enabled DPU flavor examples now mark each physical OVS uplink with the matching interface name assigned inside the HBN container by the DPUDeployment (p0 to p0h and p1 to p1h on BlueField-4; p0 to p0_if and p1 to p1_if on earlier BlueField generations). Keeping the flavor metadata and container interface names aligned allows nl2doca to identify the interfaces as uplinks and apply configured NAT rules.
    • Internal Ref #5182732

Installation & Upgrade Notes

Installation Notes

None.

Upgrade Notes

  • Pre-upgrade OpenAPI validation

    • Description: Before upgrading its managed components, the DPF operator now validates all existing API objects against the OpenAPI schema of the installed CRDs and blocks the upgrade if any violations are found. See Object Schema Validation for details and remediation steps.
    • Impact: Operators upgrading from a release where the CRD schema has tightened (e.g. a field is now required) must bring all existing resources into compliance before the upgrade can complete.

  • Kamaji prerequisite must be upgraded to chart version 1.4.0

    • Description: The kamaji cluster manager in this release uses TenantControlPlane fields that are only declared in the CRDs shipped with the kamaji Helm chart version 1.4.0 (see Helm prerequisites).
    • Impact: If the DPF operator is upgraded while an older kamaji chart (for example 1.2.0) is still installed, every DPUCluster reconcile fails with failed to upgrade TCP ... field not declared in schema until the kamaji chart is upgraded. Upgrade the kamaji prerequisite before or together with the DPF operator.

  • DPF controllers now default to 2 replicas (high availability)

    • Description: The DPF control-plane controllers managed by the operator (provisioning, dpuservice, cluster-manager, nvipam, serviceset, nodesriovdeviceplugin) now default to 2 replicas, with leader election and pod anti-affinity, so loss of the leader pod fails over in seconds instead of waiting for a pod reschedule. The dpf-operator pod itself is unchanged (1 replica).
    • Impact: Each affected controller roughly doubles its CPU and memory requests on control-plane nodes. The standby replica is passive and starts no additional kube-apiserver watches until it becomes the leader.
    • Note: The replica count is configurable per controller via DPFOperatorConfig.spec.<controller>.replicas (allowed range 1-3). To keep the previous single-replica behaviour, set it to 1, for example: 
      kubectl patch dpfoperatorconfig dpfoperatorconfig -n dpf-operator-system --type merge \
  -p '{"spec":{"dpuServiceController":{"replicas":1}}}'

  • BFB Registry external load balancer must serve HTTPS after upgrade

    • Description: Starting with this release, the bfb-registry is served exclusively over HTTPS and all clients communicate with it over HTTPS with TLS verification (see the BFB Registry now served over HTTPS feature above). If you provide your own external load balancer for the bfb-registry via DPFOperatorConfig.spec.provisioningController.registry.loadBalancerAddress, you must update that load balancer before upgrading so that:
      • The load balancer exposes an HTTPS endpoint to its clients (HostAgent / Redfish).
      • The load balancer forwards traffic to the backend bfb-registry service over HTTPS as well.
      • The load balancer's server CA certificate is added to the DPF CA trust bundle so clients can validate it. Add the PEM certificate to the dpf-ca-trust-bundle ConfigMap (data key ca.crt) in the DPF operator namespace.
    • Impact: If the load balancer is not switched to HTTPS and its CA certificate is not added to the dpf-ca-trust-bundle, provisioning traffic that flows through it (BFB download and bf.cfg fetch by the HostAgent / Redfish) fails TLS verification and provisioning cannot complete.
    • Note: Keep the existing HTTP endpoint on the load balancer in place while adding the HTTPS endpoint, so pre-upgrade clients (still using HTTP) and post-upgrade clients (using HTTPS) are both served during the rollout. Once the upgrade is complete and all clients have switched to HTTPS, the HTTP endpoint can be removed.

  • BMC factory reset applies to newly onboarded DPUs, and the ResettingBMC condition is gone

    • Description: DPUDevices onboarded after the upgrade are initialized with a one-time BMC factory reset (see the BMC factory reset and password hardening feature above). DPUDevices that DPF already manages are never reset by the upgrade: the controller recognizes them and records BMCFactoryResetReady=True with reason FactoryResetSkipped, regardless of the policy field's default value.
    • Required action before upgrade: If any BMC that will be onboarded after the upgrade has a statically configured IP address, set discoveredDPUDeviceBMCFactoryResetPolicy: Never in the DPFOperatorConfig, or create those DPUDevices with spec.bmcFactoryResetPolicy: Never. A reset erases the BMC network configuration, and such a BMC does not come back at the address in status.bmcIp.
    • Impact: The ResettingBMC DPUDevice condition has been removed along with the mTLS failure path that set it. Anything watching it must move to BMCFactoryResetReady, which has the opposite polarity: True means the reset step is finished, not that a reset is running.

  • Customer-supplied container registry proxy now requires TLS verification

    • Description: In previous releases, when DPUFlavor.spec.containerdConfig.registryEndpoint was configured, DPF set up the DPU containerd registry mirror for nvcr.io with TLS certificate verification skipped. Starting with this release, containerd validates the registry proxy server certificate against the DPF CA trust bundle. To keep the mirror working, the registry proxy's CA certificate must be added to the dpf-ca-trust-bundle ConfigMap (data key ca.crt) in the DPF operator namespace.
    • Required action before upgrade: If you use a customer-supplied container registry proxy via registryEndpoint, create (or update) the dpf-ca-trust-bundle ConfigMap before upgrading and add the registry proxy's CA certificate (PEM) to the ca.crt key. If multiple CA certificates are needed (for example the bfb-registry load balancer CA and the registry proxy CA), concatenate them into the same ca.crt value.
    • Impact: If the registry proxy CA certificate is not present in the dpf-ca-trust-bundle, containerd on the DPU fails to validate the mirror's certificate and image pulls through the proxy fail.

Known Issues and Limitations