Internal telemetry documentation and policy

July 24, 2026 · View on GitHub

Status: Draft under active development.

Scope

This documentation applies to all telemetry produced by the OTAP dataflow engine (runtime and its core libraries):

resource metadata (service, host, container, process)
metrics
events (structured logs with an event name)
traces (when implemented)

Normative language

The keywords MUST, SHOULD, and MAY are to be interpreted as normative requirements in all the documentation within this directory.

Internal telemetry is a first-class concern of this project. As with any complex system, reliable operation, performance analysis, and effective debugging require intentional and well-designed instrumentation. This document defines the principles, guidelines, and implementation details governing the project's internal telemetry.

We follow an observability by design approach: observability requirements are defined early and evolve alongside the system itself. All main entities or components are expected to be instrumented consistently, using well-defined schemas and conventions, so that emitted telemetry is coherent, actionable, and suitable for long-term analysis and optimization.

This approach is structured around the following lifecycle:

Set clear goals: Define observability objectives up front. Identify which signals are required and why.
Automate: Use tooling to derive code, documentation, tests, and schemas from shared conventions.
Validate: Detect observability and schema issues early through CI and automated checks, not in production.
Iterate: Refine telemetry based on real-world usage, feedback, and evolving system requirements.

Telemetry is treated as a stable interface of the system. As with any public API, backward compatibility, semantic clarity, and versioning discipline are essential. Changes to telemetry should be intentional, reviewed, and aligned with the overall observability model.

See the Stability and Compatibility Guide for the stability model, compatibility rules, and deprecation process.

Goals

Internal telemetry MUST enable:

reliable operation and incident response
performance analysis and regression detection
capacity planning and saturation detection
change impact analysis (deploys, config reloads, topology changes)
long-term trend analysis with stable schema and naming

Telemetry MUST NOT compromise:

system safety and correctness
performance budgets on hot paths
confidentiality (PII, secrets, sensitive payloads)

Core principles

The principles below define how internal telemetry is designed, implemented, validated, and evolved in this project. They are intentionally opinionated and serve as a shared contract between contributors, tooling, and runtime behavior.

1. Schema-first

All telemetry is defined schema-first. Entities, signals, attributes, and their relationships MUST be described explicitly in a schema before or alongside their implementation.

Schemas are treated as versioned artifacts and as the primary source of truth for:

instrumentation requirements,
validation rules,
documentation generation,
and client SDK generation.

Ad hoc or implicit telemetry definitions are discouraged, as they undermine consistency, tooling, and long-term maintainability.

2. Entity-centric

Telemetry is modeled around entities, which represent stable, identifiable subjects of observation. Signals describe the state, behavior, or performance of one or more entities at a given point in time.

This project favors:

clear separation between entity attributes (stable context) and signal-specific attributes (dynamic context),
bounded and well-justified attribute cardinality,
stable identifiers to support correlation across signals, restarts, and system boundaries.

Entity modeling is a prerequisite for producing telemetry that is interpretable, composable, and operationally useful at scale.

3. Type-safe and performance-focused instrumentation

The telemetry SDK is type-safe by construction and performance-aware.

Instrumentation APIs should:

prevent invalid or non-compliant telemetry at compile time whenever possible,
minimize overhead on hot paths,
avoid unnecessary allocations and dynamic behavior,
make the cost of instrumentation explicit and predictable.

Correctness, efficiency, and safety take precedence over convenience.

4. Runtime-safe and failure-resilient telemetry

Telemetry is designed to be runtime-safe and failure-resilient by default.

Instrumentation and telemetry pipelines MUST be non-fatal, bounded, and isolated from critical execution paths.

Specifically:

Export failures MUST NOT break or stall the dataflow engine.
Telemetry pipelines MUST use bounded buffers.
Under pressure, the default behavior SHOULD be to drop telemetry rather than block critical work.
Drops SHOULD be observable via counters, categorized by drop reason, and optionally via debug events.

The telemetry system MUST NOT introduce deadlocks, unbounded memory growth, or process termination. Similar principles have been established in the official OpenTelemetry documentation for error handling.

5. Alignment with OpenTelemetry semantic conventions

This project adopts OpenTelemetry semantic conventions as the baseline vocabulary and modeling framework.

Where existing conventions are sufficient, they are reused directly. Where project-specific concepts are required, they are defined in a custom semantic convention registry, aligned with OpenTelemetry principles and formats.

This registry formally describes:

the entities relevant to the project,
the signals emitted by the system,
the allowed attributes, types, units, and stability guarantees.

6. First-class support for multivariate metrics

The internal telemetry model and SDK natively support multivariate metric sets.

This enables:

efficient sharing of attribute tuples,
coherent modeling of related measurements,
reduced duplication explosion compared to naive univariate metrics.

Multivariate metrics are treated as a fundamental modeling capability rather than a post-processing optimization.

Note: OTLP and OTAP protocols do not yet have first-class support for multivariate metrics. The SDK and exporters handle the necessary translation and encoding. We plan to contribute multivariate support to OpenTelemetry protocols in the future. In the meantime, this project serves as a proving ground for the concept.

7. Tooling-driven validation and documentation with Weaver

Telemetry correctness and completeness are enforced through tooling, not convention alone.

This project plans to integrate with Weaver to:

validate emitted telemetry against the versioned semantic convention registry,
perform registry compliance checks in CI,
execute live checks during tests to ensure that expected signals are actually produced,
generate authoritative documentation in Markdown or HTML from the registry.

An administrative endpoint exposes the live, resolved schema at runtime to support inspection, debugging, and tooling integration.

Security and deployment guidance for this endpoint is in the Security and Privacy Guide.

Registry compliance checks and live checks are not yet enforced in CI. See Implementation Gaps.

8. Automated client SDK generation (longer term)

In the longer term, the custom semantic convention registry will be used to generate schema-driven, type-safe, and optimized Rust client SDKs via Weaver. Meanwhile, we built a type-safe Rust instrumentation API on top of the standard client SDK.

The objective is to:

eliminate manual duplication between schema and code,
ensure strict alignment between instrumentation and specification,
provide contributors with safe, ergonomic APIs that encode observability rules directly in types.

This is considered a strategic investment and will be introduced incrementally.

9. Telemetry as a stable interface

Telemetry is treated as a stable interface of the system. Refer to Stability and Compatibility Guide.

Instrumentation guides

Instrumentation is the act of adding telemetry signals (metrics, events, traces) to the codebase to observe the system behavior and performance.

The entity model defines the observed things, the "nouns" of our system, and how signals describe them. Entities are described by attributes that provide context to metrics, events, and traces, and a single signal can involve multiple entities at once. Attribute cardinality must be bounded to keep telemetry efficient and aggregations meaningful. Identifier stability matters for correlation across signals and restarts; refer to the stability guarantees in the entity model when adding new attributes.

The naming conventions, units and general guidelines are in the semantic conventions guide. Contributors SHOULD read it before introducing new telemetry.

The guides below provide a framework for defining good, consistent, secure, and actionable signals. They are not an exhaustive list of every signal and attribute in the project, but a shared reference for how to introduce and evolve telemetry:

Implementation details

For implementation details of the telemetry SDK, including macros, schema handling, and the dataflow for metric collection, see the telemetry implementation description.

Note: This SDK is internal to the project and optimized for our use cases. It is not intended for public use (at least not yet). It may change without notice.

The documentation in this directory focuses on the intented design and policy aspects of internal telemetry. The current implementation does not yet fully realize all goals and principles described here, but it is evolving rapidly. The implementation gaps document tracks the progress.

Contributor workflow

When adding or changing telemetry:

Update the semantic convention registry first (schema-first).
Regenerate documentation and code (when applicable).
Run CI validation when available (registry checks, live checks in tests).
If the change is breaking, bump the registry version and add a migration note.

Implementation status of this workflow (what is enforced, generated, and validated) is tracked in implementation-gaps.md. Coordinate with maintainers when making changes that are not yet tooling-supported.