Distributed Tracing via OpenTelemetry

Trickster instruments Distributed Tracing with OpenTelemetry. We import the OpenTelemetry golang packages to instrument support for tracing.

As OpenTelemetry evolves to support additional exporter formats, we will work to extend Trickster to support those as quickly as possible. We also make a best effort to update our otel package imports to the latest releases, whenever we publish a new Trickster release. You can check the go.mod file to see which release of opentelemetry-go we are is using. In this view, to see which version of otel a specific Trickster release imports, use the branch selector dropdown to switch to the tag corresponding to that version of Trickster.

Supported Tracing Backends

• Jaeger (via OTLP)
• Console/Stdout (printed locally by the Trickster process)

Trickster's OTLP exporter supports OTLP over HTTP and gRPC. HTTP is the default for existing OTLP configs. For Jaeger over OTLP/HTTP, configure Trickster with the collector's HTTP endpoint, for example http://jaeger:4318/v1/traces. For Jaeger over OTLP/gRPC, set protocol: grpc and use the collector's gRPC endpoint, for example http://jaeger:4317 from another Compose service or http://127.0.0.1:4317 from the host when using the developer Compose environment.

Configuration

Trickster allows the operator to configure multiple tracing configurations, which can be associated into each Backend configuration by name.

The example config has exhaustive examples of configuring Trickster for distributed tracing.

For OTLP tracing, protocol selects the exporter transport. Supported values are http and grpc; when omitted, Trickster uses http for backward compatibility with existing configs.

Context Propagation

When tracing is enabled for a Backend, Trickster uses the W3C Trace Context and Baggage propagators. It extracts incoming traceparent, tracestate, and baggage headers from client requests and injects the active outbound origin request span into the proxied request. This lets downstream Origins continue the same distributed trace instead of starting an unrelated trace.

Sampling

Trickster uses parent-based sampling for traced requests. The configured sample_rate controls root traces that Trickster starts when there is no sampled upstream trace context. When an incoming request already has a remote parent trace, Trickster follows the parent's sampling decision so sampled upstream traces continue across the proxy boundary and unsampled upstream traces remain unsampled.

Span List

Trickster can insert several spans to the traces that it captures, depending upon the type and cacheability of the inbound client request, as described in the table below.

Span Name	Observes when Trickster is:
request	initially handling the client request by a Backend
QueryCache	querying the cache for an object
WriteCache	writing an object to the cache
DeltaProxyCacheRequest	handling a Time Series-based client request
FetchFastForward	making a Fast Forward request for time series data
FetchTimeSeries	retrieving time series data from an Origin
FetchRange	retrieving one sharded time range from an Origin
Fetch	retrieving one object or range from an Origin
FetchRevalidation	revalidating a stale cache object against its Origin
ProxyRequest	communicating with an Origin server to fulfill a client request
PrepareFetchReader	preparing a client response from a cached or Origin response
CacheRevalidation	revalidating a stale cache object against its Origin
FetchObject	retrieving a non-time-series object from an Origin

Tags / Attributes

Trickster supports adding custom tags to every span via the configuration. Depending upon your preferred tracing backend, these may be referred to as attributes. See the example config for examples of adding custom attributes.

Trickster also supports omitting any tags that Trickster inserts by default. The list of default tags are below. For example on the "request" span, an http.url tag is attached with the current full URL. In deployments where that tag may introduce too much cardinality in your backend trace storage system, you may wish to omit that tag and rely on the more concise path tag. Each tracer config can be provided a string list of tags to omit from traces.

Attributes added to request and core proxy/cache/fetch spans

• backend.name
• backend.provider
• cache.name
• cache.provider
• router.path - request path trimmed to the route match path for the request (e.g., /api/v1/query), good for aggregating when there are large variations in the full URL path
• router.handler

These resource attributes are attached when the corresponding backend, cache, or route configuration is available to the request flow.

Attributes added to top level (request) span

• http.url - the full HTTP request URL

Attributes added to cache/proxy spans

• cache.status - the lookup or proxy cache status where available. See the cache status reference for a description of the attribute values.

Attributes added to proxy/fetch spans

• http.status_code - the HTTP status returned by the Origin or generated proxy response, when available.

Attributes added to the FetchRevalidation span

• isRange - is true if the client request includes an HTTP Range header

Attributes added to the FetchObject span

• isPCF - is true if the origin is configured for Progressive Collapsed Forwarding