NIXL Telemetry System

Overview

The NIXL telemetry system provides real-time monitoring and performance tracking capabilities for NIXL applications. It collects various metrics and events during runtime and exports them via an active telemetry plug-in. Only one plug-in can be loaded per NIXL instance. The Plug-In that stores telemetry data in shared memory buffer can only work as a built-in module (static linking). Other plug-ins (e.g., the one that uses Prometheus exporter) can only be linked dynamically. Custom telemetry exporter plug-ins can be created according to src/plugins/telemetry/README.md.

Architecture

Telemetry Components

1. Telemetry Collection: Built into the NIXL core library, collects events and metrics
2. Shared Memory Buffer: Statically-linked built in implementation of telemetry exporter. Uses shared memory cyclic buffer for efficient event storage and export.
3. Telemetry Readers: C++ and Python applications to read and display telemetry data from the cyclic buffer.
4. Prometheus exporter: EXPERIMENTAL (beta) Prometheus compatible telemetry exporter, see src/plugins/telemetry/prometheus/README.md.
5. DOCA exporter: EXPERIMENTAL DOCA/CollectX telemetry exporter. Drives one or more delivery backends (NIXL_TELEMETRY_DOCA_BACKENDS, default scrape): a local Prometheus scrape endpoint and/or ipc push to the DOCA Telemetry Service (DTS) for single-endpoint multi-process aggregation. See src/plugins/telemetry/doca/README.md.

Event Structure

Each telemetry event contains:

• Event type: Descriptive name/identifier for the event
• Value: Numeric value associated with the event

Event Type Naming

Event categories are not stored as a separate telemetry field. Filtering can be done by event-type prefix instead:

Event Type Prefix	Former Category	Example Events
agent_memory_	Memory	agent_memory_registered, agent_memory_deregistered
agent_tx_, agent_rx_	Transfer	agent_tx_bytes, agent_rx_requests_num
agent_xfer_	Performance	agent_xfer_time, agent_xfer_post_time
agent_err_	Error	agent_err_backend, agent_err_invalid_param

Metrics & Events Description

The following table lists all built-in telemetry metrics and events generated by NIXL:

Event Name	Unit	Description
agent_memory_registered	bytes	registered memory size per registration API call
agent_memory_deregistered	bytes	bytes of memory deregistered per API call
agent_tx_bytes	bytes	bytes transmitted by the agent per TX request
agent_rx_bytes	bytes	bytes received by the agent per RX request
agent_tx_requests_num	count	Number of transmit requests sent by the agent
agent_rx_requests_num	count	Number of receive requests processed by the agent
agent_xfer_time	microseconds	Transfer time from start to complete (per request)
agent_xfer_post_time	microseconds	Time from start to posting to backend (per request)
agent_err_*	count	Error occurrences by status type
agent_telemetry_events_dropped	count	Telemetry events dropped at the producer-side staging queue (delta per flush)

Every event value is a single per-operation reading (e.g. the bytes moved by one request), never a running total. Exporters derive two views from that same stream:

• Cumulative counters: the sum of all per-operation values over the instance lifetime, accumulated by the exporter (e.g. agent_tx_bytes_total).
• Last-operation gauges: the value of the most recent operation, re-emitted unchanged (e.g. agent_tx_last_bytes). For example, TX byte sizes 10, 20, 35 yield a counter agent_tx_bytes_total of 65 and a gauge agent_tx_last_bytes of 35. The Prometheus and DOCA exporters emit identical series -- the same names, types, and labels -- derived from one shared metric descriptor (nixlEnumStrings::telemetryMetricDescriptor in telemetry_event.h). Both expose agent_tx_bytes_total / agent_rx_bytes_total (counters, OpenMetrics _total suffix) alongside agent_tx_last_bytes / agent_rx_last_bytes (gauges). The memory events likewise expose both a cumulative _total counter and a _last_bytes gauge on both exporters, and the transfer-time events expose a _total counter alongside a last-op gauge (agent_xfer_time / agent_xfer_post_time). This is purely an exporter-side derivation: no new event type is emitted and the buffer format is unchanged.
• Latency histograms: the transfer-time events additionally feed distribution histograms agent_xfer_time_us / agent_xfer_post_time_us (microseconds) on both the Prometheus and DOCA exporters, at parity (same names, buckets, labels). Each is exposed as the standard _bucket{le="..."} / _sum / _count series alongside the existing counter and gauge. Bucket boundaries default to a microsecond range covering ~10us..~10s and are overridable via NIXL_TELEMETRY_HISTOGRAM_BUCKETS_US (a comma-separated list of strictly-increasing positive microsecond upper bounds; when absent or empty the built-in defaults are used, while a non-empty but invalid value is rejected and the exporter fails to initialize rather than silently using the defaults). Like the other views this is an exporter-side derivation with no new event type.
• Error counters: the Prometheus and DOCA exporters expose error events as agent_errors_total{status="<status>"}. The status label is bounded by the fixed AGENT_ERR_* event set: not_posted, invalid_param, backend, not_found, mismatch, not_allowed, repost_active, unknown, not_supported, remote_disconnect, canceled, and no_telemetry.
• Dropped-events counter: agent_telemetry_events_dropped is a cumulative counter of telemetry events lost at the producer-side staging queue. When a producer cannot enqueue an event because the staging queue is full (updateData, or the whole 4-event addXferStats batch), the event is dropped and counted. On each flush the core emits the count of new drops since the last flush as a synthetic AGENT_TELEMETRY_EVENTS_DROPPED event, which flows through the shared descriptor as agent_telemetry_events_dropped_total on both exporters and into the raw BUFFER stream. This counts only staging-queue loss (events that never reach an exporter); it does not count BUFFER cyclic-ring loss from a slow downstream reader, which is a separate, uncounted condition.

Metric naming convention

These conventions follow Prometheus/OpenMetrics and apply to the byte metrics described above; they are the target for new metrics:

• The base unit is the terminal suffix, in base units (_bytes, _seconds).
• Cumulative counter: agent_<subject>_<unit>_total (e.g. agent_tx_bytes_total). _total is reserved for counters; gauges never end in _total.
• Last-operation gauge: agent_<subject>_last_<unit> (e.g. agent_tx_last_bytes) -- the _last qualifier precedes the unit, so the name stays convention-compliant (unit last) and sorts next to its sibling counter.
• The metric type (counter vs gauge) is conveyed by the # TYPE metadata (# HELP is descriptive only), never encoded in the name apart from the conventional _total.

Known exceptions (predating this convention): the transfer-time metrics agent_xfer_time_total / agent_xfer_post_time_total report microseconds and do not carry a base-unit suffix; aligning them is a separate change.

The Shared Memory Buffer plug-in, contains the data per transaction event, without summarizing between events.

Telemetry Details

• Producers append events under a mutex, consumers read them in ring insertion order.
• Current design allows silent telemetry loss.
• Current design does not support selective telemetry. All the telemetry events could be either ON or OFF.

Enabling Telemetry

Runtime Configuration

Telemetry is configured by environment variables:

Variable	Description	Default
NIXL_TELEMETRY_ENABLE	Enable telemetry collection	false
NIXL_TELEMETRY_BUFFER_SIZE	Number of events in buffer	4096
NIXL_TELEMETRY_RUN_INTERVAL	Flush interval (ms)	100
NIXL_TELEMETRY_EXPORTER	Name of the exporter plugin to use	-
NIXL_TELEMETRY_HISTOGRAM_BUCKETS_US	Comma-separated microsecond bucket bounds for the transfer-time histograms (Prometheus/DOCA)	built-in µs defaults
NIXL_TELEMETRY_ENABLED_METRICS	Comma-separated allowlist of metric names to export (glob)	all

• NIXL_TELEMETRY_ENABLE can be set to y/yes/on/true/enable/1 to be enabled, and n/no/off/false/disable/0 (or not set) to be disabled. Matching is case insensitive.
• NIXL_TELEMETRY_ENABLED_METRICS restricts which metrics are exported. It is a comma-separated allowlist of metric names, each matched as a POSIX glob (*, ?) against the base event names (agent_tx_bytes, agent_rx_bytes, agent_tx_requests_num, agent_rx_requests_num, agent_memory_registered, agent_memory_deregistered, agent_xfer_time, agent_xfer_post_time, agent_telemetry_events_dropped, and the agent_err_* errors). Unset (or empty) exports everything; a token that matches nothing is ignored with a warning. A metric name selects every series derived from that event (counter and gauge, plus the transfer-time histograms agent_xfer_time_us / agent_xfer_post_time_us where applicable). Deactivated metrics are skipped at the source, before entering the staging queue, so they add no cost on the transfer hot path. Per-transfer stats returned by getXferTelemetry() come from the request handle and are unaffected.
• Telemetry is requested either via NIXL_TELEMETRY_ENABLE or via the agent config flag captureTelemetry (capture_telemetry=True in Python). It is fully off only when it is not requested.
• When telemetry is requested but no output sink is configured (neither NIXL_TELEMETRY_EXPORTER nor NIXL_TELEMETRY_DIR), it falls back to the collect-only NOP exporter: events are collected in-process so getXferTelemetry() / get_xfer_telemetry() works, but nothing is written out.
• If telemetry is enabled but no exporter is set, or the exporter name is empty, then the sink depends on NIXL_TELEMETRY_DIR as explained below (falling back to NOP when it is unset).
• Set NIXL_TELEMETRY_EXPORTER=NOP to explicitly keep telemetry active (events are collected and getXferTelemetry() works) while discarding all output. It needs no sink and writes nothing, so it can be used to measure the overhead of the telemetry collection path in isolation.
• Exporters that expose a scrape endpoint (e.g. Prometheus) bind one port per process. Under multi-process runs (e.g. tensor/data parallelism) every rank tries to bind the same port; only one wins. Losing that race is benign and non-fatal: the affected process logs a single warning and runs without a telemetry sink instead of failing agent construction. See src/plugins/telemetry/prometheus/README.md.

Cyclic Buffer

Following sections applied specifically for configuration and usage of cyclic buffer exporter

Configuration

Variable	Description	Default
NIXL_TELEMETRY_DIR	Directory for telemetry files	-

• If telemetry is requested but NIXL_TELEMETRY_DIR is not set (and no other exporter is selected), no telemetry file is generated; telemetry instead falls back to the collect-only NOP exporter, so events are collected in-process (getXferTelemetry() works) but nothing is written and NIXL_TELEMETRY_RUN_INTERVAL has no observable effect.

Telemetry File Format

Telemetry data is stored in shared memory files with the agent name passed when creating the agent.

TELEMETRY_VERSION is the binary layout version of nixlTelemetryEvent as serialized in the shared-memory cyclic buffer. Bump it whenever the struct's size, field offsets, or enum widths change, or when a new event type is added to the stream (readers must know the full event-type set). Readers exact-match this value; the C++ shared-ring-buffer path unlinks the file on mismatch, while the Python example raises a RuntimeError. There is no backward-compatibility path. The current version is 4: it was bumped from 3 when the agent_telemetry_events_dropped event type (AGENT_TELEMETRY_EVENTS_DROPPED) was added.

Using Telemetry Readers

C++ Telemetry Reader

The C++ telemetry reader (telemetry_reader.cpp) provides a robust way to read and display telemetry events.

Running the C++ Reader

# Read from a specific telemetry file
./builddir/examples/cpp/telemetry_reader /tmp/agent_name

Python Telemetry Reader

The Python telemetry reader (telemetry_reader.py) provides similar functionality with additional features.

Running the Python Reader

# Read from a specific telemetry file
python3 examples/python/telemetry_reader.py --telemetry_path /tmp/agent_name

Example Output

Both readers produce similar formatted output:

=== NIXL Telemetry Event ===
Event: agent_tx_bytes
Value: 1048576
===========================

=== NIXL Telemetry Event ===
Event: agent_memory_registered
Value: 4096
===========================