This document outlines all Prometheus metrics exposed by NVSentinel components.
| Metric Name | Type | Labels | Description |
|---|
fault_quarantine_events_received_total | Counter | - | Total number of events received from the watcher |
fault_quarantine_events_successfully_processed_total | Counter | - | Total number of events successfully processed |
fault_quarantine_processing_errors_total | Counter | error_type | Total number of errors encountered during event processing |
fault_quarantine_cold_start_events_total | Counter | result | Health events examined during startup recovery, labeled processed, skipped, superseded, invalid, or failed |
fault_quarantine_cold_start_duration_seconds | Histogram | - | Startup recovery duration. Buckets: ExponentialBuckets(start=0.1s, factor=2, count=18), max ~3.6 hours |
fault_quarantine_event_backlog_count | Gauge | - | Number of health events which fault quarantine is yet to process |
fault_quarantine_event_handling_duration_seconds | Histogram | - | Histogram of event handling durations |
| Metric Name | Type | Labels | Description |
|---|
fault_quarantine_nodes_quarantined_total | Counter | node | Total number of nodes quarantined |
fault_quarantine_nodes_unquarantined_total | Counter | node | Total number of nodes unquarantined |
fault_quarantine_nodes_manually_uncordoned_total | Counter | node | Total number of manually uncordons for nodes |
fault_quarantine_nodes_manually_untainted_total | Counter | node | Total number of manual taint removals for nodes |
fault_quarantine_current_quarantined_nodes | Gauge | node | Current number of quarantined nodes |
| Metric Name | Type | Labels | Description |
|---|
fault_quarantine_taints_applied_total | Counter | taint_key, taint_effect | Total number of taints applied to nodes |
fault_quarantine_taints_removed_total | Counter | taint_key, taint_effect | Total number of taints removed from nodes |
fault_quarantine_cordons_applied_total | Counter | - | Total number of cordons applied to nodes |
fault_quarantine_cordons_removed_total | Counter | - | Total number of cordons removed from nodes |
fault_quarantine_node_quarantine_duration_seconds | Histogram | - | Time from health event generation to node quarantine completion. Buckets: Prometheus DefBuckets |
fault_quarantine_node_remediation_duration_seconds | Histogram | recommended_action | End-to-end node remediation time: generatedTimestamp (from original unhealthy event) to node unquarantine. Emitted on both auto unquarantine (via healthy event) and manual uncordon. Buckets: ExponentialBuckets(start=10s, factor=1.5, count=27), max ~4.4 days |
fault_quarantine_node_remediation_duration_excluding_drain_seconds | Histogram | recommended_action | Remediation time excluding node-drainer duration: (unquarantineTime - generatedTimestamp) - (drainFinishTimestamp - quarantineFinishTimestamp). Emitted only when both quarantineFinishTimestamp and drainFinishTimestamp are present in the original event document. Buckets: ExponentialBuckets(start=10s, factor=1.5, count=19), max ~4.1 hours |
| Metric Name | Type | Labels | Description |
|---|
fault_quarantine_ruleset_evaluations_total | Counter | ruleset, status | Total number of ruleset evaluations. Status values: passed, failed |
| Metric Name | Type | Labels | Description |
|---|
fault_quarantine_breaker_state | Gauge | state | State of the fault quarantine breaker |
fault_quarantine_breaker_utilization | Gauge | - | Fraction of GPU nodes cordoned within the circuit breaker's sliding window |
fault_quarantine_breaker_threshold_nodes | Gauge | bound | Cordoned-node count that trips the circuit breaker, and which configured bound produced it (percentage, maxNodes, or fleetSize when clamped) |
fault_quarantine_get_total_nodes_duration_seconds | Histogram | result | Duration of getTotalNodesWithRetry calls in seconds |
fault_quarantine_get_total_nodes_errors_total | Counter | error_type | Total number of errors from getTotalNodesWithRetry |
fault_quarantine_get_total_nodes_retry_attempts | Histogram | - | Number of retry attempts needed for getTotalNodesWithRetry (buckets: 0, 1, 2, 3, 5, 10) |
| Metric Name | Type | Labels | Description |
|---|
node_drainer_events_received_total | Counter | - | Total number of events received from the watcher |
node_drainer_events_replayed_total | Counter | - | Total number of in-progress events replayed at startup |
node_drainer_events_processed_total | Counter | drain_status, node | Total number of events processed by drain status outcome. Drain status values: drained, cancelled, skipped |
node_drainer_cancelled_event_total | Counter | node, check_name | Total number of cancelled drain events (due to manual uncordon or healthy recovery) |
node_drainer_processing_errors_total | Counter | error_type, node | Total number of errors encountered during event processing and node draining |
node_drainer_event_handling_duration_seconds | Histogram | - | Histogram of event handling durations |
node_drainer_queue_depth | Gauge | - | Total number of pending events in the queue |
node_drainer_pod_eviction_duration_seconds | Histogram | - | Time from event receipt by node-drainer to successful pod eviction completion. Buckets: Exponential (0.1s, factor 2, 23 buckets, up to ~3 days) |
| Metric Name | Type | Labels | Description |
|---|
node_drainer_waiting_for_timeout | Gauge | node | Shows if node drainer operation is waiting for timeout before force deletion (1=waiting, 0=not waiting) |
node_drainer_force_delete_pods_after_timeout | Counter | node, namespace | Total number of node drainer operations that reached timeout and force deleted pods |
| Metric Name | Type | Labels | Description |
|---|
fault_remediation_events_received_total | Counter | - | Total number of events received from the watcher |
fault_remediation_events_processed_total | Counter | cr_status, node_name | Total number of remediation events processed by CR creation status. CR status values: created, skipped |
fault_remediation_processing_errors_total | Counter | error_type, node_name | Total number of errors encountered during event processing |
fault_remediation_unsupported_actions_total | Counter | action, node_name | Total number of health events with currently unsupported remediation actions |
fault_remediation_event_handling_duration_seconds | Histogram | - | Histogram of event handling durations |
fault_remediation_cr_generate_duration_seconds | Histogram | - | Time from drain completion (or quarantine completion if drain timestamp unavailable) to maintenance CR creation. Buckets: Prometheus DefBuckets |
| Metric Name | Type | Labels | Description |
|---|
fault_remediation_log_collector_jobs_total | Counter | node_name, status | Total number of log collector jobs. Status values: success, failure, timeout |
fault_remediation_log_collector_job_duration_seconds | Histogram | node_name, status | Duration of log collector jobs in seconds |
fault_remediation_log_collector_errors_total | Counter | error_type, node_name | Total number of errors encountered in log collector operations |
| Metric Name | Type | Labels | Description |
|---|
http_response_count_total | Counter | method, status, app | Total HTTP responses by method and status code |
http_request_duration_seconds | Histogram | method, status | HTTP request duration in seconds |
| Metric Name | Type | Labels | Description |
|---|
fileserver_log_rotation_successful_total | Counter | - | Total successful log cleanup operations |
fileserver_log_rotation_failed_total | Counter | - | Total failed log cleanup operations |
fileserver_disk_space_free_bytes | Gauge | - | Free disk space in bytes |
| Metric Name | Type | Labels | Description |
|---|
labeler_events_processed_total | Counter | status | Total number of pod events processed. Status values: success, failed |
labeler_node_update_failures_total | Counter | - | Total number of node update failures during reconciliation |
labeler_event_handling_duration_seconds | Histogram | - | Histogram of event handling durations |
| Metric Name | Type | Labels | Description |
|---|
janitor_actions_count | Counter | action_type, status, node | Total number of janitor actions by type and status. Action types: reboot, terminate. Status values: started, succeeded, failed |
janitor_action_mttr_seconds | Histogram | action_type | Time from CR creation to action completion (Mean Time To Repair). Uses exponential buckets (10, 2, 10) for log-scale MTTR measurement |
These track the node-binding interceptor that stops a caller on one node from
submitting health events naming another node. See
Health Event Authentication.
| Metric Name | Type | Labels | Description |
|---|
platform_connector_auth_decisions_total | Counter | decision | Health event batches by the node scope granted to the caller. Values: node_local, cross_node |
platform_connector_auth_node_claim_total | Counter | result | Whether an authenticated caller's token carried a node claim. verified: it carried one and it named this node (on the deployment platform connector, which has no node of its own, the claim is taken as the caller's node). absent: it carried none, so the caller was pinned to this node, the same scope a tokenless caller gets; on the deployment platform connector such a caller is rejected instead (node_claim_absent or unbound_token; cross_node_claim_absent or unbound_cross_node_token when the identity is on the cross-node list). |
platform_connector_auth_violations_total | Counter | reason | Health event batches that violated the node-binding rule. Under global.platformConnectorAuth.mode: enforce (default) these are rejected; under mode: audit they are recorded here but let through. Reasons: node_mismatch (a node-local caller named a different node), node_claim_mismatch (the token was issued on another node; applies to every caller, allowlisted or not), unbound_cross_node_token (an allowlisted caller presented a token bound to no pod, e.g. from kubectl create token), cross_node_claim_absent (pod-bound, but the pod was never scheduled), missing_node_name (no node name on the event and none could be stamped), token_invalid (TokenReview rejected the token), malformed_credentials (the authorization header was duplicated or did not use the Bearer scheme). Three reasons occur only on the deployment platform connector, which has no local node to fall back to: token_missing (no token at all), unbound_token (a token bound to no pod), node_claim_absent (a token bound to a pod that never scheduled, so no node to pin its events to). Three further reasons mean no verdict could be reached, not that the caller was rejected: validator_unavailable (the API server was unreachable), validator_timeout (the caller gave up or its deadline passed) and validator_error (an unexpected failure, typically this component's own RBAC to create TokenReviews). With failOpenOnUnavailable: true, validator_unavailable and validator_timeout still increment this counter but fall back to a degraded node-local scope instead of rejecting the request, in mode: enforce too, because an outage says nothing about the caller's credential. That scope is a guess, not a verified identity: an event with a blank or matching node name is accepted and stamped as usual in either mode, but one naming a different node depends on mode: under enforce it is refused as retryable Unavailable rather than counted as node_mismatch, since the caller might really be an allowlisted cross-node publisher the outage prevented from being verified; under audit that same Unavailable is recorded and then let through like any other audited violation, so the event is forwarded with whatever node name it carried. missing_node_name is the one violation mode: audit does not let through: a blank node name from a caller resolved to verified cross-node scope is always rejected, in both modes, because nothing downstream can handle it. This does not cover a caller that fell back to a degraded node-local scope under failOpenOnUnavailable: its identity was never verified as cross-node, so a blank name from it is stamped like any other node-local caller's. Exclude those three when alerting on suspected credential abuse, because a control-plane outage increments them for every in-flight request and would otherwise look identical to an attack. The selector to alert on is below this table. |
The alerting selector for suspected credential abuse, with the three no-verdict reasons left out:
sum(rate(platform_connector_auth_violations_total{reason=~"node_mismatch|token_invalid|node_claim_mismatch|unbound_cross_node_token|malformed_credentials|cross_node_claim_absent|missing_node_name|token_missing|unbound_token|node_claim_absent"}[5m]))
| Metric Name | Type | Labels | Description |
|---|
k8s_platform_connector_node_condition_update_total | Counter | status | Total number of node condition updates by status. Status values: success, failed, skipped (an update that would leave the node's conditions as they are, such as a monitor's repeat of a fault the node already shows, is skipped) |
k8s_platform_connector_node_event_operations_total | Counter | operation, status | Total number of node event operations by type and status. Operation values: create, update. Status values: success, failed, skipped (a repeat of a fault whose Event was written less than 10 minutes ago is skipped; later repeats refresh the Event) |
k8s_platform_connector_node_condition_update_duration_milliseconds | Histogram | - | Duration of node condition updates in milliseconds. Uses linear buckets (0, 10, 500) |
k8s_platform_connector_node_event_update_create_duration_milliseconds | Histogram | - | Duration of node event updates/creations in milliseconds. Uses linear buckets (0, 10, 500) |
Every health monitor publishes through the platform connector, so this one metric covers
gpu-health-monitor, syslog-health-monitor, nic-health-monitor, csp-health-monitor,
kubernetes-object-monitor and health-events-analyzer without per-monitor instrumentation.
Enabled with platformConnector.promConnector.enabled (default false, like the other optional connectors).
| Metric Name | Type | Labels | Description |
|---|
health_events_total | Counter | node, agent, check_name, recommended_action, is_fatal, is_healthy | Health events received by the platform connector. recommended_action is the enum name (NONE, CONTACT_SUPPORT, RESTART_VM, …); a CUSTOM action reports as CUSTOM rather than expanding customRecommendedAction, which is operator-defined and unbounded |
node carries the event's own nodeName, not the node of the connector pod that
received it. That distinction matters: the DaemonSet monitors publish over the node-local
socket so the two coincide, but health-events-analyzer is a Deployment whose derived
events describe other nodes, and an allowlisted cross-node publisher can name any node. An
explicit label makes every agent correct without a kube_pod_info join.
errorCode is excluded because it is unbounded: suffixed XIDs such as
145.RLW_SRC_TRACK would grow the series set without limit. node is bounded by the fleet.
is_fatal is kept even though producers derive it as recommendedAction != NONE. The
redundancy is the point: a producer that disagrees with that derivation becomes visible
rather than silent.
Useful queries:
# Actionable event rate by action, fleet-wide
sum by (recommended_action) (rate(health_events_total{recommended_action!="NONE"}[5m]))
# Which agent and check is producing the actionable events
sum by (agent, check_name) (rate(health_events_total{recommended_action!="NONE",is_healthy="false"}[1h]))
# Per node, correct for every agent including health-events-analyzer
sum by (node) (rate(health_events_total{recommended_action!="NONE"}[1h]))
Both roles of the platform connector, the node-local DaemonSet and the deployment platform connector, expose these for the batches that reach the request handler. Alert on failed outcomes and on best-effort failures. The deployment platform connector (--mode=deployment) also exposes the metrics above for its Kubernetes connector and for the node-binding interceptor (its rejections are counted in platform_connector_auth_violations_total, by the reasons listed there), and the refusals below, which it answers before the handler sees the batch.
| Metric Name | Type | Labels | Description |
|---|
platform_connector_request_duration_seconds | Histogram | outcome | Duration of health event batch requests that reached the handler, by outcome: ok (acknowledged), rejected (the batch is invalid), failed (the connector returned an error while processing the batch; the caller retries). Both roles expose it: on the node-local DaemonSet the connector is the set of ring buffers, which always accept, so a batch is ok once queued unless it is invalid; on the deployment platform connector the store connector's write decides when config.json enables one, while the node update and the sink are best effort and never fail a batch, so without a store every batch that passed validation is ok. |
platform_connector_deployment_refusals_total | Counter | reason | Batches refused before the handler saw them: index_unverified (this replica has a store connector and has not verified the idempotency index; the caller retries) or idempotency_key (the idempotency-key header is missing or malformed) |
platform_connector_store_batches_total | Counter | outcome | Batches written to the datastore by either role: stored, or duplicate (a resend, or a retried batch, whose events already existed; treated like a store) |
platform_connector_best_effort_failures_total | Counter | connector, reason | Deployment platform connector only: work a best-effort connector (kubernetes node conditions and Events, grpcsink forwards) did not finish although the batch was acknowledged: failed or timeout; the node shows the old state until the monitor next reports |
These metrics track the internal ring buffer workqueue performance:
| Metric Name | Type | Labels | Description |
|---|
platform_connector_workqueue_depth_{name} | Gauge | workqueue | Current depth of Platform connector workqueue |
platform_connector_workqueue_adds_total_{name} | Counter | workqueue | Total number of adds handled by Platform connector workqueue |
platform_connector_workqueue_latency_seconds_{name} | Histogram | workqueue | How long an item stays in Platform connector workqueue before being requested. Uses linear buckets (0, 10, 500) |
platform_connector_workqueue_work_duration_seconds_{name} | Histogram | workqueue | How long processing an item from Platform connector workqueue takes. Uses linear buckets (0, 10, 500) |
platform_connector_workqueue_retries_total_{name} | Counter | workqueue | Total number of retries handled by Platform connector workqueue |
platform_connector_workqueue_longest_running_processor_seconds_{name} | Gauge | workqueue | How many seconds the longest running processor for Platform connector workqueue has been running |
platform_connector_workqueue_unfinished_work_seconds_{name} | Gauge | workqueue | The total time in seconds of work in progress in Platform connector workqueue |
Note: {name} in the metric names is replaced with the actual workqueue name at runtime.
The Go health monitors send their events through the shared publishing client (commons/pkg/healthpub). The monitor label names the monitor. The socket rows apply to the node-local path; the direct-mode rows apply only when a monitor publishes to the deployment platform connector (HEALTH_PUBLISH_TARGET set, which the chart's publishTo: deployment does once the chart ships).
| Metric Name | Type | Labels | Description |
|---|
nvsentinel_health_events_publisher_skipped_pc_unavailable_total | Counter | monitor | Socket-path sends skipped because the node-local platform connector socket was missing |
nvsentinel_health_events_publisher_sends_success_total | Counter | monitor | Batches the platform connector acknowledged, on either path |
nvsentinel_health_events_publisher_sends_error_total | Counter | monitor, code | Socket-path sends that failed after retries, by gRPC status code |
nvsentinel_health_events_publisher_dropped_total | Counter | monitor, reason | Direct-mode batches given up on: rejected (the server refuses the batch for good, or it exceeds the 4 MiB message limit), retry_window_exhausted, shutdown, withdrawn (the caller stopped waiting: its context ended or its timeout expired, and the batch was taken back) |
nvsentinel_health_events_publisher_retries_total | Counter | monitor | Direct-mode send attempts that failed and were retried inside the batch's retry window |
These metrics track GPU health events detected via DCGM (Data Center GPU Manager):
| Metric Name | Type | Labels | Description |
|---|
dcgm_health_events_publish_time_to_grpc_channel | Histogram | operation_name | Amount of time spent in publishing DCGM health events on the gRPC channel |
health_events_insertion_to_uds_succeed | Counter | - | Total number of successful insertions of health events to UDS |
health_events_insertion_to_uds_error | Counter | - | Total number of failed insertions of health events to UDS |
health_events_insertion_skipped_pc_unavailable | Counter | - | Sends skipped because the node-local platform connector socket was missing |
health_events_direct_publish_succeed | Counter | - | Batches the deployment platform connector acknowledged (direct mode) |
health_events_direct_publish_dropped | Counter | reason | Direct-mode batches given up on; the same reasons as the Go publisher's dropped_total |
health_events_direct_publish_retries | Counter | - | Direct-mode send attempts that failed and were retried |
dcgm_health_active_events | Gauge | event_type, gpu_id, error_code | Total number of active GPU health events at any given time |
dcgm_health_active_switch_events | Gauge | event_type, switch_id, error_code | Total number of active NVSwitch health events at any given time. Separate from dcgm_health_active_events because switch and GPU ids are distinct sequences that both start at 0 |
dcgm_api_latency | Histogram | operation_name | Amount of time spent calling DCGM APIs |
dcgm_reconcile_time | Histogram | - | Amount of time spent running a single DCGM reconcile loop |
number_of_health_watches | Gauge | - | Number of DCGM health watches available |
number_of_fields | Gauge | - | Number of available DCGM fields to monitor |
callback_failures | Counter | class_name, func_name | Number of times a callback function has thrown an exception |
callback_success | Counter | class_name, func_name | Number of times a callback function has successfully completed |
dcgm_api_failures | Counter | error_name | Number of DCGM API errors |
dcgm_health_check_unknown_system_skipped | Counter | - | Number of DCGM health check incidents skipped due to unrecognized system value |
dcgm_probe_hangs | Counter | operation_name | Number of DCGM probes that exceeded the watchdog deadline without returning |
The syslog health monitor tracks GPU-related errors detected from system logs.
XID (GPU Error ID) errors are NVIDIA GPU driver errors:
| Metric Name | Type | Labels | Description |
|---|
syslog_health_monitor_xid_errors | Counter | node, err_code | Total number of XID errors found |
syslog_health_monitor_xid_processing_errors | Counter | error_type, node | Total number of errors encountered during XID processing |
syslog_health_monitor_xid_processing_latency_seconds | Histogram | - | Histogram of XID processing latency |
SXID errors are NVSwitch-related errors:
| Metric Name | Type | Labels | Description |
|---|
syslog_health_monitor_sxid_errors | Counter | node, err_code, link, nvswitch | Total number of SXID errors found |
| Metric Name | Type | Labels | Description |
|---|
syslog_health_monitor_gpu_fallen_errors | Counter | node | Total number of GPU fallen off bus errors detected |
The CSP health monitor tracks cloud provider maintenance events and node health issues.
| Metric Name | Type | Labels | Description |
|---|
csp_health_monitor_csp_events_received_total | Counter | csp | Total number of raw events received from CSP API/source |
csp_health_monitor_csp_polling_duration_seconds | Histogram | csp | Duration of CSP polling cycles |
csp_health_monitor_csp_api_errors_total | Counter | csp, error_type | Total number of errors encountered during CSP API calls |
csp_health_monitor_csp_api_polling_duration_seconds | Histogram | csp, api | Duration of CSP API polling cycles |
csp_health_monitor_csp_monitor_errors_total | Counter | csp, error_type | Total number of errors initializing or starting CSP monitors |
csp_health_monitor_csp_events_by_type_unsupported_total | Counter | csp, event_type | Total number of raw CSP events received, partitioned by event type code |
| Metric Name | Type | Labels | Description |
|---|
csp_health_monitor_main_events_to_normalize_total | Counter | csp | Total number of events passed to the normalizer |
csp_health_monitor_main_normalization_errors_total | Counter | csp | Total number of errors during event normalization |
csp_health_monitor_main_events_received_total | Counter | csp | Total number of normalized events received by the main processor |
csp_health_monitor_main_events_processed_success_total | Counter | csp | Total number of events successfully processed |
csp_health_monitor_main_processing_errors_total | Counter | csp, error_type | Total number of errors during event processing |
csp_health_monitor_main_event_processing_duration_seconds | Histogram | csp | Duration of processing a single event |
| Metric Name | Type | Labels | Description |
|---|
csp_health_monitor_main_datastore_upsert_attempts_total | Counter | csp | Total number of attempts to upsert maintenance events |
csp_health_monitor_main_datastore_upsert_total | Counter | csp, status | Total number of maintenance event upserts by status. Status values: success, failed |
| Metric Name | Type | Labels | Description |
|---|
csp_health_monitor_trigger_poll_cycles_total | Counter | - | Total number of polling cycles executed by the trigger engine |
csp_health_monitor_trigger_poll_errors_total | Counter | - | Total number of errors during a trigger engine poll cycle |
csp_health_monitor_trigger_events_found_total | Counter | trigger_type | Total number of events found potentially needing a trigger |
csp_health_monitor_trigger_attempts_total | Counter | trigger_type | Total number of trigger attempts made |
csp_health_monitor_trigger_success_total | Counter | trigger_type | Total number of successful triggers |
csp_health_monitor_trigger_failures_total | Counter | trigger_type, failure_reason | Total number of failed trigger attempts |
csp_health_monitor_trigger_datastore_query_duration_seconds | Histogram | query_type | Duration of datastore queries performed by the trigger engine |
csp_health_monitor_trigger_datastore_query_errors_total | Counter | query_type | Total number of errors during datastore queries |
csp_health_monitor_trigger_datastore_update_errors_total | Counter | trigger_type | Total number of errors updating event status after trigger |
csp_health_monitor_trigger_uds_send_duration_seconds | Histogram | - | Duration of sending health events via UDS |
csp_health_monitor_trigger_uds_send_errors_total | Counter | - | Total number of errors encountered when sending events via UDS |
csp_health_monitor_node_not_ready_timeout_total | Counter | node_name | Total number of nodes that remained not ready after the timeout period |
csp_health_monitor_node_readiness_monitoring_started_total | Counter | node_name | Total number of times background node readiness monitoring was started |
Emitted by store-client, so they appear on every module that reads the change stream:
event-exporter, fault-quarantine, health-events-analyzer, node-drainer, and
fault-remediation, on both the MongoDB and PostgreSQL providers. The client label is the
consumer's name.
fault-remediation serves only controller-runtime's registry, so it passes that registry
explicitly; the others use the default registry.
| Metric Name | Type | Labels | Description |
|---|
change_stream_lag_seconds | Gauge | client | Seconds since this consumer last had evidence it was caught up with its own change stream, from either an empty batch or the server-side timestamp of the last event it read. Absent until one of those has been observed |
change_stream_lag_known | Gauge | client | 1 once change_stream_lag_seconds can be computed for this consumer, 0 before then |
change_stream_resume_token_recoveries_total | Counter | client, phase | Times a stale or invalid resume token was deleted, restarting the stream from the current position |
Lag is measured against the consumer's own filtered stream, so a module whose pipeline
admits nothing still reports zero lag while it is caught up. See
ADR-054.
# A consumer is behind. Tune the threshold per fleet; no lag_known qualifier is needed,
# because the series is absent rather than zero while lag is unknown.
- alert: ChangeStreamConsumerBehind
expr: change_stream_lag_seconds > 900
for: 10m
# A watcher that never started, or is wedged before its first read. The grace period covers
# normal startup, which closes in milliseconds on a live stream.
- alert: ChangeStreamLagUnknown
expr: change_stream_lag_known == 0
for: 10m
Each of these is a way a consumer can be behind while the gauge reads zero.
- Replication lag. The MongoDB stream is opened
SecondaryPreferred with no max staleness,
so an empty batch means "caught up with the secondary I am reading", not with the primary.
- Data skipped after a resume-token recovery. When a stored token is too old for the oplog
the watcher reopens from now, so lag reads near zero precisely when the most was skipped.
Read
change_stream_lag_seconds together with change_stream_resume_token_recoveries_total:
a lag of zero is only reassuring if the recoveries counter has not moved.
- Durable position. This measures the watcher's progress against its own stream, not
whether its position was persisted. A consumer that reads an event and dies before
MarkProcessed succeeds looks healthy here, because the read did happen.
All NVSentinel components expose Prometheus metrics on a metrics endpoint (typically :2112/metrics). The metrics can be scraped by Prometheus using standard scrape configurations.
The NVSentinel Helm chart automatically creates a PodMonitor resource for Prometheus Operator integration:
helm install nvsentinel ./distros/kubernetes/nvsentinel \
--namespace nvsentinel --create-namespace
The PodMonitor is configured to scrape all NVSentinel component pods on their metrics endpoints (/metrics on port metrics).
Components can be configured to include Prometheus scrape annotations:
annotations:
prometheus.io/scrape: "true"
prometheus.io/port: "2112"
prometheus.io/path: "/metrics"
- Counter: A cumulative metric that only increases or resets to zero on restart
- Gauge: A metric that can arbitrarily go up and down
- Histogram: Samples observations and counts them in configurable buckets
- Summary: Similar to histogram but calculates configurable quantiles over a sliding time window
success / failed - Operation outcomestarted / succeeded / failed - Action lifecycle status
reboot - Node reboot actionterminate - Node termination action
gcp - Google Cloud Platformaws - Amazon Web Services
quarantine - Node quarantine triggerhealthy - Node healthy trigger