NIXL Custom Telemetry Plugin Development Guide

This guide explains how to create custom telemetry exporter plugins for NIXL. Telemetry plugins allow you to export NIXL telemetry data to different monitoring systems, databases, or file formats.

Overview

NIXL telemetry plugins are dynamically loaded shared libraries that export telemetry events from the NIXL agent

Built-in Event Types

NIXL generates the following telemetry events:

Event Name	Type	Description
agent_memory_registered	Gauge	Total bytes of memory registered
agent_memory_deregistered	Gauge	Total bytes of memory deregistered
agent_tx_bytes	Counter	Total bytes transmitted
agent_rx_bytes	Counter	Total bytes received
agent_tx_requests_num	Counter	Number of transmit requests
agent_rx_requests_num	Counter	Number of receive requests
agent_xfer_time	Counter, Gauge, Histogram	Transfer time (start->completion) in microseconds; also a latency histogram agent_xfer_time_us
agent_xfer_post_time	Counter, Gauge, Histogram	Post time (start->backend-post) in microseconds; also a latency histogram agent_xfer_post_time_us
agent_telemetry_events_dropped	Counter	Telemetry events dropped at the producer-side staging queue

The NIXL_TELEMETRY_ENABLED_METRICS environment variable is a comma-separated glob allowlist of the event names above, plus the built-in agent_err_* error events (unset exports everything). Events whose name is not activated are skipped at the source before they enter the staging queue, so they never reach the exporter and cost nothing on the hot path.

Quick Start

Here's a minimal example of a CSV file exporter plugin:

1. Create Your Exporter Class (csv_exporter.h)

#ifndef NIXL_TELEMETRY_CSV_EXPORTER_H
#define NIXL_TELEMETRY_CSV_EXPORTER_H

#include "telemetry/telemetry_exporter.h"
#include <fstream>

class nixlTelemetryCsvExporter : public nixlTelemetryExporter {
public:
    explicit nixlTelemetryCsvExporter(const nixlTelemetryExporterInitParams *init_params);
    ~nixlTelemetryCsvExporter() override;

    nixl_status_t exportEvent(const nixlTelemetryEvent &event) override;

private:
    std::ofstream file_;
};

#endif // _TELEMETRY_CSV_EXPORTER_H

2. Implement Your Exporter (csv_exporter.cpp)

#include "csv_exporter.h"
#include "common/nixl_log.h"
#include "common/configuration.h"

nixlTelemetryCsvExporter::nixlTelemetryCsvExporter(
    const nixlTelemetryExporterInitParams *init_params)
    : nixlTelemetryExporter(init_params) {

    auto file_path = nixl::config::getValue<std::string>("NIXL_TELEMETRY_CSV_FILE");
    file_.open(file_path, std::ios::out | std::ios::app);
    if (!file_.is_open()) {
        throw std::runtime_error("Failed to open CSV file: " + file_path);
    }

    // Write CSV header
    file_ << "event_type,value\n";
    NIXL_INFO << "CSV exporter initialized: " << file_path;
}

nixl_status_t
nixlTelemetryCsvExporter::exportEvent(const nixlTelemetryEvent &event) {
    if (!file_.is_open()) {
        return NIXL_ERR_UNKNOWN;
    }

    try {
        file_ << nixlEnumStrings::telemetryEventTypeStr(event.eventType_) << "," << event.value_
              << "\n";
        file_.flush();
        return NIXL_SUCCESS;
    }
    catch (const std::exception &e) {
        NIXL_ERROR << "Failed to export event: " << e.what();
        return NIXL_ERR_UNKNOWN;
    }
}

3. Create Plugin Interface (csv_plugin.cpp)

#include "csv_exporter.h"
#include "telemetry/telemetry_plugin.h"

// Use the plugin creator template for minimal boilerplate
using csv_exporter_plugin_t = nixlTelemetryPluginCreator<nixlTelemetryCsvExporter>;

// Plugin initialization function - must be extern "C"
extern "C" NIXL_TELEMETRY_PLUGIN_EXPORT nixlTelemetryPlugin *
nixl_telemetry_plugin_init() {
    return csv_exporter_plugin_t::create(
        nixl_telemetry_plugin_api_version::V2,
        "csv",      // Plugin name
        "1.0.0"     // Plugin version
    );
}

// Plugin cleanup function
extern "C" NIXL_TELEMETRY_PLUGIN_EXPORT void
nixl_telemetry_plugin_fini() {
    // Add any global cleanup if needed
}

4. Build Configuration (meson.build)

# CSV Exporter Plugin
csv_exporter_plugin = shared_library(
    'libtelemetry_exporter_csv',
    'csv_plugin.cpp',
    'csv_exporter.cpp',
    include_directories: [nixl_inc_dirs, utils_inc_dirs],
    dependencies: [nixl_infra, absl_log_dep],
    install: true,
    install_dir: get_option('libdir') / 'nixl' / 'telemetry_exporters',
    name_prefix: '',
)