Setting up Monitoring Stack on Kubernetes

October 14, 2025 · View on GitHub

This guide helps you set up Prometheus and Grafana on vanilla Kubernetes to visualize Kepler metrics.

Overview

Kepler exports metrics in Prometheus format. To use Kepler Operator, you need:

prometheus-operator - REQUIRED for ServiceMonitor support (Kepler Operator creates ServiceMonitor resources)
Prometheus - Optional, to collect and store Kepler metrics for visualization
Grafana - Optional, to visualize metrics with dashboards

Important: Kepler Operator requires prometheus-operator to be installed because it creates ServiceMonitor custom resources. You can install just prometheus-operator without Prometheus/Grafana, but you won't be able to collect or visualize metrics.

What Do You Need?

Required: prometheus-operator

You MUST install prometheus-operator before installing Kepler Operator. The operator will fail to reconcile PowerMonitor resources without it.

Optional: Full Monitoring Stack

You need a complete monitoring stack (Prometheus + Grafana) if you want to:

Visualize energy consumption metrics in Grafana dashboards
Query historical power consumption data
Set up alerts based on energy usage
Integrate with existing monitoring infrastructure

You can install only prometheus-operator without Prometheus/Grafana, but Kepler metrics won't be collected or visualized.

Quick Start: kube-prometheus-stack (Recommended)

The easiest way to set up monitoring on Kubernetes is using the kube-prometheus-stack Helm chart, which includes:

Prometheus Operator
Prometheus instance
Grafana
AlertManager
Pre-configured dashboards and alerts

Install kube-prometheus-stack

# Add Helm repository
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update

# Install the stack
helm install prometheus prometheus-community/kube-prometheus-stack \
  --namespace monitoring \
  --create-namespace \
  --set prometheus.prometheusSpec.serviceMonitorSelectorNilUsesHelmValues=false

# Wait for all monitoring components to be ready
kubectl wait --for=condition=ready --timeout=180s pod -n monitoring --all

Important: The serviceMonitorSelectorNilUsesHelmValues=false setting allows Prometheus to discover ServiceMonitors from all namespaces, not just those created by this Helm release.

Verify Installation

# Check pods are running
kubectl get pods -n monitoring

# Check Prometheus is accessible
kubectl port-forward -n monitoring svc/prometheus-kube-prometheus-prometheus 9090:9090
# Open http://localhost:9090 in your browser

# Check Grafana is accessible
kubectl port-forward -n monitoring svc/prometheus-grafana 3000:80
# Open http://localhost:3000 in your browser
# Default credentials: admin / prom-operator

Configure Kepler Integration

When installing Kepler Operator via Helm, enable ServiceMonitor creation:

helm install kepler-operator ./manifests/helm/kepler-operator \
  --namespace kepler-operator \
  --create-namespace \
  --set metrics.serviceMonitor.enabled=true

This creates a ServiceMonitor that Prometheus will automatically discover.

Alternative: Manual Setup

If you prefer not to use the kube-prometheus-stack, you can install components individually.

Install prometheus-operator

kubectl create -f https://github.com/prometheus-operator/prometheus-operator/releases/download/v0.76.0/bundle.yaml

Install Prometheus Instance

Create a Prometheus instance that watches for ServiceMonitors:

# prometheus-instance.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: prometheus
  namespace: monitoring
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: prometheus
rules:
- apiGroups: [""]
  resources:
  - nodes
  - nodes/metrics
  - services
  - endpoints
  - pods
  verbs: ["get", "list", "watch"]
- apiGroups: [""]
  resources:
  - configmaps
  verbs: ["get"]
- nonResourceURLs: ["/metrics"]
  verbs: ["get"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: prometheus
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: prometheus
subjects:
- kind: ServiceAccount
  name: prometheus
  namespace: monitoring
---
apiVersion: monitoring.coreos.com/v1
kind: Prometheus
metadata:
  name: prometheus
  namespace: monitoring
spec:
  serviceAccountName: prometheus
  serviceMonitorSelector: {}  # Select all ServiceMonitors
  resources:
    requests:
      memory: 400Mi
  enableAdminAPI: false

Apply the configuration:

kubectl create namespace monitoring
kubectl apply -f prometheus-instance.yaml

Install Grafana

Follow the official Grafana documentation to install Grafana on Kubernetes:

Grafana Installation Guide

Or use the Grafana Helm chart:

helm repo add grafana https://grafana.github.io/helm-charts
helm repo update

helm install grafana grafana/grafana \
  --namespace monitoring \
  --set persistence.enabled=true \
  --set adminPassword=admin

Configure Prometheus as Grafana Data Source

Access Grafana (default port 3000)
Navigate to Configuration → Data Sources
Click Add data source
Select Prometheus
Set URL to: http://prometheus-operated.monitoring.svc:9090
Click Save & Test

Verification

Check Prometheus is Scraping Kepler

After installing Kepler Operator and creating a PowerMonitor:

Access Prometheus UI (port-forward to port 9090)
Navigate to Status → Targets
Look for kepler targets - they should be in "UP" state

Alternatively, query for Kepler metrics:

kepler_node_cpu_joules_total

If you see results, Prometheus is successfully scraping Kepler metrics.

Check ServiceMonitor

Verify the ServiceMonitor was created:

kubectl get servicemonitor -A | grep power-monitor

Next Steps

Now that your monitoring stack is ready:

Install Kepler Operator with ServiceMonitor enabled
Create a PowerMonitor to deploy Kepler
Import Grafana dashboards to visualize metrics

Troubleshooting

Prometheus Not Discovering ServiceMonitor

If Prometheus isn't discovering the Kepler ServiceMonitor:

Check if prometheus-operator is running:

kubectl get pods -n monitoring | grep prometheus-operator

Check Prometheus configuration for ServiceMonitor selector:
```
kubectl get prometheus -n monitoring -o yaml | grep -A5 serviceMonitorSelector
```
If it has specific label selectors, your ServiceMonitor must match those labels.

Check ServiceMonitor labels:

kubectl get servicemonitor power-monitor -n power-monitor -o yaml

No Kepler Metrics in Prometheus

If Kepler targets appear in Prometheus but no metrics are available:

Check Kepler pods are running:

kubectl get pods -n power-monitor -l app.kubernetes.io/name=power-monitor-exporter

Verify Kepler is exposing metrics:

kubectl port-forward -n power-monitor daemonset/power-monitor 28282:28282
curl http://localhost:28282/metrics | grep kepler

Check ServiceMonitor configuration matches Kepler service:

kubectl get svc power-monitor -n power-monitor
kubectl get servicemonitor power-monitor -n power-monitor -o yaml