Monitoring Kubernetes Clusters

May 30, 2024 ยท View on GitHub

Monitoring your Kubernetes cluster lets you see its health and performance. Statistics such as CPU, memory, and disk usage are available for both Linux and Windows nodes in your AKS Engine cluster.

Resource metrics are collected by the lightweight, in-memory metrics-server component. Metrics-server discovers nodes and queries each one's kubelet for CPU and memory usage.

Metrics-server is part of every AKS Engine deployment. To verify that metrics-server is running in your cluster:

$ kubectl get pods --namespace=kube-system -l k8s-app=metrics-server
NAME                             READY   STATUS    RESTARTS   AGE
metrics-server-bb7db87bc-nm6vn   1/1     Running   2          140m

Tools like kubectl and the Kubernetes Dashboard use metrics-server, and it has an API to get metrics for your custom monitoring solution.

Monitoring Options

  1. kubectl
  2. Kubernetes Dashboard
  3. Monitoring extension

Kubectl

The kubectl top command is an easy way to see node or pod metrics in your terminal.

Use kubectl top node to see the resource consumption of nodes:

$ kubectl top node
NAME                                 CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%
k8s-agentpool1-26399701-vmss000000   67m          3%     729Mi           10%
k8s-agentpool1-26399701-vmss000001   80m          4%     787Mi           10%
k8s-master-26399701-0                201m         10%    1406Mi          19%

Use kubectl top pod to see the resource consumption of pods:

$ kubectl top pod --namespace=kube-system
NAME                                            CPU(cores)   MEMORY(bytes)
azure-cni-networkmonitor-7gfd4                  2m           15Mi
...
kube-proxy-mzlq5                                1m           18Mi
kube-scheduler-k8s-master-26399701-0            3m           16Mi
metrics-server-bb7db87bc-nm6vn                  1m           12Mi

Kubernetes Dashboard

The Kubernetes Dashboard is a web-based user interface that can visualize cluster metrics.

Describing all of the useful ways to use the dashboard project is out of scope of this documentation. See here to learn more.

Monitoring extension

A quick way to scaffold out cloud-native and open source monitoring components is to use the aks-engine-azurestack monitoring extension. For details on how to use the monitoring extension, please refer to the extension documentation. By embedding the extension in your apimodel, the extension will do much of the work to create a monitoring solution in your cluster, which includes the following:

  • cAdvisor daemon set to publish container metrics
  • Prometheus for metrics collection and storage
  • Grafana for dashboard and visualizations

The extension wires up these components together. Post-deployment of the Kubernetes cluster, you just have to retrieve Grafana admin password (Kubernetes secret) and target your browser to the Grafana endpoint. There is already a pre-loaded Kubernetes cluster monitoring dashboard, so out-of-the-box you will have meaningful monitoring points with the extensibility that Prometheus and Grafana offer you.