KEDA GPU Scaler

August 14, 2026 · View on GitHub

Scale Kubernetes GPU workloads from real hardware metrics. No DCGM. No PromQL. Optional Prometheus metrics built in.

A KEDA External Scaler that reads NVIDIA GPU metrics directly from NVML C-bindings and autoscales your vLLM, Triton, and custom inference deployments including scale-to-zero.

Why This Exists

Kubernetes HPA watches CPU and memory. It can't see GPU utilization. Your vLLM pod shows 8% CPU while the GPU is at 100%.

The usual fix is dcgm-exporter → Prometheus → KEDA, but that's 5 components and 15-30s of latency.

This project reads GPU metrics directly from NVML and serves them to KEDA over gRPC. 2 components, 2-4 second latency.

Why Not a Native KEDA Scaler?

Putting GPU support inside KEDA core doesn't work:

CGO Constraint: NVIDIA's Go bindings (go-nvml) require CGO_ENABLED=1. KEDA builds with CGO_ENABLED=0.
Node-Level Hardware Access: The KEDA operator runs as a central pod. NVML requires local GPU device access via libnvidia-ml.so, which only a DaemonSet on GPU nodes can provide.
Independent Release Cycle: Ship GPU scaling improvements without waiting for KEDA release cycles.

This design is documented in KEDA issue #7538.

Architecture

Detailed Data Flow

keda-gpu-scaler data flow

DaemonSet — Runs on nodes labeled with nvidia.com/gpu.present: "true".
NVML Bindings — Directly reads Streaming Multiprocessor (SM) utilization and Frame Buffer Memory via go-nvml C-bindings.
gRPC Interface — Implements externalscaler.ExternalScalerServer (IsActive, StreamIsActive, GetMetricSpec, GetMetrics) to natively integrate with the central KEDA operator.
ScaledObject Trigger — Kubernetes deployments scale up/down (including to zero) based on GPU thresholds defined in the ScaledObject.

⚠️ Supported Topology: Each DaemonSet pod reports metrics for its local node only. Multi-node cluster-wide aggregation is not yet implemented — for multi-GPU-node clusters, deploy one ScaledObject per node or use node selectors. See #142 for the roadmap.

GPU Metrics

Metric	Description	Unit
`gpu_utilization`	GPU compute (SM) utilization	% (0-100)
`memory_utilization`	GPU memory controller utilization	% (0-100)
`memory_used_mib`	GPU VRAM used	MiB
`memory_used_percent`	GPU VRAM used as percentage of total	% (0-100)
`temperature`	GPU die temperature	Celsius
`power_draw`	GPU power consumption	Watts
`pcie_tx_kbps`	PCIe transmit throughput (CPU→GPU)	KB/s
`pcie_rx_kbps`	PCIe receive throughput (GPU→CPU)	KB/s
`nvlink_tx_mbps`	NVLink transmit throughput (GPU→GPU)	MB/s
`nvlink_rx_mbps`	NVLink receive throughput (GPU→GPU)	MB/s
`vllm_queue_depth`	Pending requests waiting in the vLLM engine — requires `vllmEndpoint`	count
`vllm_kv_cache_usage`	vLLM GPU KV cache usage — requires `vllmEndpoint`	% (0-100)
`triton_queue_wait_ms`	Average Triton inference queue wait time — requires `tritonEndpoint`	ms
`triton_request_rate`	Triton inference request rate — requires `tritonEndpoint`	requests/sec

The vllm_* metrics come from the vLLM engine's own /metrics endpoint rather than NVML — see docs/configuration.md. The triton_* metrics likewise come from Triton's own /metrics endpoint and are derived from its cumulative counters — see docs/configuration.md.

Pre-built Scaling Profiles

Instead of configuring raw metric thresholds, use a profile optimized for your workload:

Profile	Primary Metric	Target	Activation	Use Case
`vllm-inference`	Memory %	80	5	vLLM / LLM serving with scale-to-zero
`vllm-queue-depth`	Pending requests	5	1	vLLM — scale on queue depth via the engine API for faster reaction time
`triton-inference`	GPU Util	75	10	NVIDIA Triton Inference Server
`triton-queue-wait`	Queue wait (ms)	50	5	Triton — scale on average inference queue wait time via the engine API
`triton-request-rate`	Requests/sec	50	1	Triton — scale on inference request rate via the engine API
`training`	GPU Util	90	0	Training jobs (no scale-to-zero)
`batch`	Memory %	70	1	Batch inference with aggressive scale-down
`ollama`	Memory %	70	3	Ollama LLM serving with scale-to-zero
`tgi-inference`	Memory %	75	5	HuggingFace TGI serving with scale-to-zero
`distributed-training`	NVLink TX	50000	5000	Data-parallel training on NVLink systems

Prerequisites

A Kubernetes cluster (e.g., OKE, GKE, EKS, AKS) with NVIDIA GPU worker nodes
KEDA v2.10+ installed in the cluster
NVIDIA GPU drivers and Device Plugin installed

Quick Start

1. Deploy the Scaler

Deploy the DaemonSet and gRPC service into your cluster. (Ensure KEDA is already installed.)

kubectl apply -f deploy/manifests.yaml

This deploys a DaemonSet that runs on every GPU node in your cluster, plus a ClusterIP Service for KEDA to discover it.

Or use Helm.

From the published OCI chart (recommended — replace <X.Y.Z> with the latest release):

helm install keda-gpu-scaler \
  oci://ghcr.io/pmady/charts/keda-gpu-scaler --version <X.Y.Z> \
  --namespace keda --create-namespace

From a local checkout:

helm install keda-gpu-scaler deploy/helm/keda-gpu-scaler \
  --namespace keda \
  --set nodeSelector."nvidia\.com/gpu\.present"=true

See the chart README for all configurable values.

2. Attach to your AI Workload

Create a ScaledObject pointing to the external scaler service:

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: vllm-inference-scaler
  namespace: ai-workloads
spec:
  scaleTargetRef:
    name: vllm-deepseek-deployment
  minReplicaCount: 1
  maxReplicaCount: 50
  triggers:
    - type: external
      metadata:
        scalerAddress: "keda-gpu-scaler.keda.svc.cluster.local:6000"
        targetGpuUtilization: "80"

Or use a pre-built profile:

triggers:
  - type: external
    metadata:
      scalerAddress: "keda-gpu-scaler.keda.svc.cluster.local:6000"
      profile: "vllm-inference"

3. Custom Configuration

Override any profile default or use raw GPU metrics directly:

triggers:
  - type: external
    metadata:
      scalerAddress: "keda-gpu-scaler.keda.svc.cluster.local:6000"
      metricType: "gpu_utilization"
      targetValue: "85"
      activationThreshold: "10"
      gpuIndex: "0"              # specific GPU index, or omit for all
      aggregation: "max"         # max, min, avg, sum, p95, p99 across GPUs
      cooldownSeconds: "120"    # suppress repeat scale-downs; 0 disables

See deploy/examples/ for ready-to-use ScaledObject manifests.

Configuration Reference

Parameter	Description	Default
`profile`	Pre-built scaling profile name	(none)
`metricType`	GPU metric to scale on	`gpu_utilization`
`targetValue`	Target metric value for scaling	`80`
`targetGpuUtilization`	Shorthand for GPU utilization target	(none)
`targetMemoryUtilization`	Shorthand for VRAM utilization target	(none)
`activationThreshold`	Value below which scale-to-zero activates	`0`
`gpuIndex`	Specific GPU index to monitor. Must be `-1` (all GPUs) or `>= 0`; other negative values are rejected	`-1` (all GPUs)
`aggregation`	Multi-GPU aggregation: `max`, `min`, `avg`, `sum`, `p95`, `p99`	`max`
`pollIntervalSeconds`	Metric polling interval	`10`
`cooldownSeconds`	Suppress repeat scale-downs for this many seconds after a scale-down. Scale-up is never delayed. `0` disables	`60`

Prometheus Metrics (Optional)

The scaler exposes an optional Prometheus-compatible /metrics endpoint for monitoring the scaler itself and GPU fleet health. This is independent of the KEDA scaling path — scaling works identically with or without it.

Enable/Disable

# Enabled by default on port 9090
--metrics-port=9090

# Disable entirely (zero overhead)
--metrics-port=0

Helm:

metrics:
  enabled: true   # set to false to disable
  port: 9090

Exposed Metrics

Metric	Type	Description
`keda_gpu_scaler_gpu_utilization_percent`	Gauge	GPU compute utilization (per GPU)
`keda_gpu_scaler_gpu_memory_used_bytes`	Gauge	GPU memory in use (per GPU)
`keda_gpu_scaler_gpu_memory_total_bytes`	Gauge	Total GPU memory (per GPU)
`keda_gpu_scaler_gpu_temperature_celsius`	Gauge	GPU temperature (per GPU)
`keda_gpu_scaler_gpu_power_draw_watts`	Gauge	GPU power draw (per GPU)
`keda_gpu_scaler_collections_total`	Counter	Total NVML collection calls
`keda_gpu_scaler_collection_errors_total`	Counter	Failed NVML collection calls
`keda_gpu_scaler_collection_duration_seconds`	Histogram	NVML collection latency
`keda_gpu_scaler_scaler_requests_total`	Counter	gRPC requests by method
`keda_gpu_scaler_scaler_request_errors_total`	Counter	gRPC errors by method

All per-GPU metrics are labeled with gpu_index, gpu_uuid, and gpu_name.

Kubernetes Probes

The scaler supports two probe mechanisms; the Helm chart defaults to gRPC.

gRPC health check (recommended)

The gRPC server implements the gRPC Health Checking Protocol (google.golang.org/grpc/health, grpc_health_v1) on the server-wide ("") service. A background checker polls NVML (DeviceCount()) every --health-check-interval (default 30s) and reports:

SERVING while NVML responds normally.
NOT_SERVING if the NVML call fails.

--health-check-interval=30s

Kubernetes 1.24+ can probe this natively:

livenessProbe:
  grpc:
    port: 6000
readinessProbe:
  grpc:
    port: 6000

Helm sets this up automatically (probes.type: grpc, the default).

HTTP probes (legacy / pre-1.24 clusters)

A dedicated HTTP probe port also exposes:

/healthz returns 200 while the process is alive.
/readyz returns 200 after NVML initializes and the first metrics collection succeeds.

--probe-port=8081

Helm:

probes:
  enabled: true
  type: http
  port: 8081

Build it Yourself

This project requires CGO_ENABLED=1 to compile the NVIDIA C-bindings.

Note

The compiled binaries (keda-gpu-scaler and gpu-metrics) dynamically link NVIDIA's NVML library and load libnvidia-ml.so at runtime. They will fail to start on any machine that does not have the NVIDIA driver installed (which provides libnvidia-ml.so) — for example, a laptop or CI runner with no NVIDIA GPU. You can still build, lint, and run the test suite without a GPU, since the tests use a mock collector (see Can I run this without a GPU?).

# Build KEDA scaler binary (requires CGO for NVML)
make build

# Build standalone GPU metrics CLI (no KEDA/gRPC needed)
make build-metrics

# Build all binaries
make build-all

# Run unit tests
make test

# Run linter
make lint

# Generate protobuf Go code
make proto

# Build and push a release image
make docker-release VERSION=v0.1.0

# Deploy to cluster
make deploy

Checking the Version

Both binaries accept a --version flag (and a bare version argument) that prints the version, Go version, and build date, then exits. Unlike normal operation, this does not require a GPU or the NVIDIA driver:

keda-gpu-scaler --version    # keda-gpu-scaler v0.5.0 (go1.26.4, built 2026-06-25)
gpu-metrics --version        # gpu-metrics v0.5.0 (go1.26.4, built 2026-06-25)

make build stamps the version from git describe at link time; builds without ldflags (e.g. go run) report dev.

Standalone GPU Metrics CLI

Collect GPU metrics without Kubernetes — works on bare metal, SLURM jobs, Flux jobs, Kubernetes pods, and Singularity containers. The same binary and the same JSON schema work everywhere.

Important

gpu-metrics requires libnvidia-ml.so (installed with the NVIDIA driver) on the host. On a machine without an NVIDIA driver it exits immediately with nvml init failed.

gpu-metrics                       # one-shot table output (env auto-detected)
gpu-metrics --format json         # JSON for scripting
gpu-metrics --format csv          # CSV for analysis
gpu-metrics --interval 5s         # continuous collection
gpu-metrics --device 0 --quiet    # single GPU, no logs
gpu-metrics --env slurm           # force environment (auto|k8s|slurm|flux|standalone)
gpu-metrics --version             # print version and exit (no GPU/NVML required)

The --env flag auto-detects the orchestrator by default. Detection priority: SLURM → Flux → Kubernetes → standalone.

Every environment emits the same unified JSON schema with an environment block so you can compare GPU performance across on-prem and cloud with identical tooling:

{
  "environment": { "orchestrator": "slurm", "node": "compute-01", "job_id": "123", "task_rank": 0 },
  "driver_version": "535.104.05",
  "collected_at": "2026-06-17T10:00:00Z",
  "devices": [...]
}

SLURM — auto-detected when SLURM_JOB_ID is set; collects only the GPUs assigned to your job step:

srun --gres=gpu:2 gpu-metrics --format json

Flux — auto-detected when FLUX_JOB_ID is set; collects only the GPUs in CUDA_VISIBLE_DEVICES:

flux run -N1 -g2 gpu-metrics --format json

Or let Flux start and stop collection automatically for the lifetime of a job via the gpu-monitor shell plugin drop-in: flux run -o gpu-monitor -N2 -g1 ./train.sh.

See HPC & Cross-Environment Metrics for full usage, and Cross-Environment Comparison Guide for comparing on-prem vs cloud GPU runs.

Or build the Docker image directly:

docker build -t your-registry/keda-gpu-scaler:v0.1.0 .
docker push your-registry/keda-gpu-scaler:v0.1.0

End-to-End GPU Tests

make test runs the unit suite against a mock collector (no GPU needed). The end-to-end suite is different: it provisions a real GPU cluster on AWS EKS, Azure AKS, or Google GKE with Terratest, installs the NVIDIA GPU operator + KEDA + this scaler, asserts the demo app scales up under GPU load and back down when idle, then tears everything down. The stacks live in infra/terraform/ and the Go suite in tests/terratest/.

⚠️ These tests provision real, billed GPU hardware (~$0.55–1.20/hr per cloud). They are manual-only and each per-cloud job is gated behind a GitHub Environment so a reviewer must approve the spend before it runs.

Running from GitHub Actions

Go to the repo's Actions tab and pick the workflow, then Run workflow:

Workflow	What it does	Inputs
E2E Cloud Tests (Apply and Destroy)	Provisions the cluster, asserts scale up/down, then destroys it.	`clouds`: `aws` \| `azure` \| `gcp` \| `all` · `confirm_cost`: type `apply` to confirm billing
E2E Plan (Manual)	Credential-light `terraform plan` + Infracost estimate — no cluster, no billing.	`clouds`: which stack to plan
E2E Destroy (Manual)	Safety net to tear down a cluster a failed run left behind.	`cloud` + exact `cluster_name` + type `destroy`

Choosing clouds: the clouds input on the apply workflow fans out to one job per cloud. Pick a single cloud to validate one provider, or all to run AWS, Azure, and GCP in parallel (each still needs its own reviewer approval and GPU quota).

Reading the results

Green check on the run = the full contract passed: the scaler DaemonSet came up, the ScaledObject went Ready, the demo app scaled 1 → N under load and N → 1 when idle.
Every run uploads an e2e-diagnostics-<cloud>-<run_id> artifact (via if: always()) containing the go test log; on failure it also includes a cluster dump (nodes, scaler pods/logs, ScaledObject, events).
The cluster auto-destroys, so to poke a live one, run the suite locally.

Running locally

Requires the cloud CLI authenticated, a remote state backend (see each stack's bootstrap/), and GPU quota in your target region:

cd tests/terratest
E2E_AWS_STATE_BUCKET=<bucket> \
  go test -tags e2e_aws -v -timeout 90m -run TestAWSGPUScalerE2E ./...

Full prerequisites (OIDC/WIF federation, required secrets/variables, GPU quota per cloud) are documented in tests/terratest/README.md.

How It Compares

	keda-gpu-scaler	dcgm-exporter + Prometheus	Custom Metrics API
Components	1 DaemonSet (+ optional /metrics)	dcgm-exporter + Prometheus + adapter	Custom metrics server
Metric latency	Sub-second (direct NVML)	15-30s (scrape interval)	Depends on implementation
Scale-to-zero	Yes (KEDA native)	Yes (with KEDA Prometheus scaler)	Manual
Configuration	3-line ScaledObject	PromQL query per metric	Custom code
GPU metrics	10 hardware metrics	50+ DCGM metrics	Whatever you build
Dependencies	KEDA, NVIDIA drivers	KEDA, Prometheus, dcgm-exporter	Varies
Failure domain	Node-local	Centralized Prometheus	Varies

Documentation

Full documentation — hosted on Read the Docs
Design Document — Architecture decisions, gRPC interface, scaling profiles, testing strategy
Migration Guide — Replace dcgm-exporter + Prometheus with keda-gpu-scaler
HPC & Cross-Environment Metrics — SLURM, Flux, Kubernetes, and standalone GPU metrics
Cross-Environment Comparison — Compare GPU performance across on-prem and cloud
Troubleshooting — Common deployment, NVML, gRPC, metric, and MIG issues
FAQ — Common questions about GPU scaling, MIG, multi-GPU, scale-to-zero
Changelog — Release history

CNCF Blog: GPU Autoscaling on Kubernetes with KEDA
KEDA issue #7538 — original discussion
CNCF TOC initiative #2188 — whitepaper proposal

Adopters

Using keda-gpu-scaler? Add your organization to ADOPTERS.md.

Roadmap

AMD ROCm support
MIG per-instance metrics
~~vLLM queue depth scaling~~ — available via the vllm_queue_depth metric type / vllm-queue-depth profile; see docs/configuration.md