Monitoring KUDO Cassandra

This guide explains how to set up monitoring for KUDO Cassandra.

Description

The KUDO Cassandra operator can export metrics to Prometheus. It achieves this using a Prometheus exporter based on the criteo/cassandra_exporter.

When the PROMETHEUS_EXPORTER_ENABLED parameter is set to true:

• A prometheus-exporter container will run in the same pod as every Cassandra node container. It will listen for connections on PROMETHEUS_EXPORTER_PORT, which is set to 7200 by default.
• A prometheus-exporter-port will be added to the KUDO Cassandra operator Service.
• A ServiceMonitor will be created to make Prometheus poll that port for metrics.

Pre-conditions

• KUDO Cassandra instance running
• Prometheus operator and Grafana set up in the cluster. The kube-prometheus project provides both of them.
• KUDO CLI installed.

The examples below assume that the instance and namespace names are stored in the following shell variables. With this assumptions met, you should be able to copy-paste the commands easily.

instance_name=cassandra
namespace_name=default

Steps

1. Make sure that Prometheus Exporter is enabled on the KUDO Cassandra instance

This parameter is false by default, so you need to enable it explicitly.

You can check the value of the parameter on a running instance with a command like:

kubectl get instance --template '{{.spec.parameters.PROMETHEUS_EXPORTER_ENABLED}}{{"\n"}} $instance_name -n $namespace_name'

An output of true means that the exporter is enabled.

Any other output means that the exporter is disabled. In that case you need to enable it with a command such as the following. If you need customization, see other parameters that start with PROMETHEUS_EXPORTER_.

kubectl kudo update -p PROMETHEUS_EXPORTER_ENABLED=true --instance $instance_name -n $namespace_name

Expected output:

Instance cassandra was updated.

2. Install the Grafana dashboard

A sample grafana dashboard is provided in the monitoring directory.

How you access the Grafana UI depends on how it was installed. Upon accessing the /dashboard/import URI you will be able to upload or copy-paste the cassandra.json file:

Once done, you will be able to see various Cassandra metrics in the dashboard:

Notes

:warning: Depending on how your prometheus operator was deployed, you may need to check the Prometheus resource. The serviceMonitorNamespaceSelector and serviceMonitorSelector attributes on that resource need to be configured to match the labels on the ServiceMonitor resource created by the KUDO Cassandra operator.

The Prometheus exporter container that is run alongside each Cassandra node requires 1 CPU and 512MiB memory each.

Custom Configuration

To use the custom prometheus exporter configuration, we need to create a configmap with the properties we want to override.

Example custom configuration:

apiVersion: v1
kind: ConfigMap
metadata:
  name: custom-exporter-configuration
data:
  config.yml: |
    maxScrapFrequencyInSec:
      2000:
        - .*:totaldiskspaceused:.*

Create the ConfigMap in the namespace we will have the KUDO Cassandra cluster

$ kubectl create -f custom-exporter-configuration.yaml -n $namespace_name
configmap/custom-exporter-configuration created

Enable the exporter

kubectl kudo update \
  -p PROMETHEUS_EXPORTER_ENABLED=true \
  -p PROMETHEUS_EXPORTER_CUSTOM_CONFIG_CM_NAME=custom-exporter-configuration \
  --instance $instance_name -n $namespace_name

:warning: The following properties are configured internally by the operator and cannot be overridden using custom configuration:

• host
• listenAddress
• listenPort
• user
• password
• ssl