Tracing Internals

September 17, 2026 · View on GitHub

Fullsend's distributed tracing system records structured telemetry for every agent run. This document explains how the tracing implementation works and how to extend it. It is aimed at contributors modifying the telemetry package or the span instrumentation in the run command.

For enabling tracing on an installation, see How To Emit Traces. For span schemas and attribute definitions, see the Tracing Reference. The design is specified in ADR 0050.

How the tracing system is structured

The tracing code is split between two locations:

internal/telemetry/ - the TracerProvider, exporters, and W3C traceparent helpers.
internal/cli/run.go - span creation, attribute assignment, and trace context propagation to child scripts.

The telemetry package owns the provider and exporters. run.go owns the span lifecycle: it decides when spans start and end, and which attributes they carry.

Package layout

internal/telemetry/
├── telemetry.go          Setup, TracerProvider wiring, env parsing
├── fileexporter.go       Synchronous OTLP JSON file exporter
├── trace.go              W3C traceparent formatting helpers
├── main_test.go
├── telemetry_test.go
├── fileexporter_test.go
├── otlpsink_test.go
└── trace_test.go

TracerProvider setup

telemetry.Setup(dir, version) returns a trace.Tracer and a cleanup function. It creates a TracerProvider with two span processors:

fileExporter via SimpleSpanProcessor: writes every completed span as one OTLP JSON line to run-telemetry.jsonl. Calls f.Sync() after each write so spans survive a process crash.
otlptracehttp via BatchSpanProcessor wrapped in parentSampledProcessor: exports to a remote backend when an OTEL_EXPORTER_OTLP_*ENDPOINT env var is set.

If neither exporter can be created (bad directory, SDK disabled), Setup returns a noop tracer. Telemetry failures never affect the run.

telemetry.Setup()
│
├── OTEL_SDK_DISABLED check   → noop tracer if "true"
├── os.OpenFile(jsonl)        → noop tracer on error
│
├── SimpleSpanProcessor(fileExporter)           ← always present
│
└── OTEL_EXPORTER_OTLP_*ENDPOINT check
    ├── validateEndpoints()   → stderr warning, skip
    ├── newOTLPExporter()     → stderr warning, skip
    └── parentSampledProcessor(BatchSpanProcessor(otlpExporter))

Why parentSampledProcessor exists

The OTel SDK's AlwaysSample sampler records all spans locally, which is needed for the file exporter. But AlwaysSample ignores an upstream unsampled decision. The parentSampledProcessor wraps the OTLP batch processor and suppresses the entire trace from export when the root span's remote parent has the W3C sampled flag unset (-00). It tracks suppressed trace IDs in a sync.Map so child spans under the same trace are also dropped.

Result: the file exporter always writes all spans; the OTLP exporter respects upstream sampling.

Endpoint validation

validateEndpoints rejects non-http(s) URLs and unsupported protocols before creating the exporter. A malformed endpoint produces a stderr warning; the SDK's default localhost:4318 fallback is never used silently.

Span lifecycle in run.go

run.go creates four span types arranged in a parent-child hierarchy:

run (root)
├── sandbox_create    (gen_ai.operation.name=create_agent)
└── agent             (one per iteration; gen_ai.operation.name=invoke_agent)
    └── execute_tool  (one per id-bearing tool call; gen_ai.operation.name=execute_tool)

Root span

Created by resolveTraceIdentity(). When an inbound TRACEPARENT env var is present, the W3C propagator extracts a remote span context and the root span is SpanKindConsumer. Otherwise it is SpanKindInternal.

Start attributes include: fullsend.agent, fullsend.work_item_id, gen_ai.operation.name, gen_ai.agent.name.

End attributes (set in a deferred cleanup): exit_code, fullsend.cost_usd, fullsend.num_turns, fullsend.tool_calls, fullsend.iterations.

For the full attribute list (including fullsend.security_trace_id and fullsend.prescript.*), see the Tracing reference attribute table.

sandbox_create span

Started before sandbox creation, ended after bootstrap completes. Carries gen_ai.operation.name=create_agent.

agent spans

One per iteration (validation loop iterations included). Started before Exec(), ended after output extraction.

The helper functions agentSpanStartAttrs() and agentSpanEndAttrs() build the attribute slices. Start attributes: iteration, gen_ai.operation.name, gen_ai.agent.name. End attributes: iteration, exit_code, gen_ai.system and gen_ai.provider.name (same serving-endpoint value), model, token counts, fullsend.cost_usd, fullsend.runtime, fullsend.tool_calls. Multi-provider runtimes resolve the provider from the effective model via runtime.GenAISystemFor; System() is only the fallback.

execute_tool spans

One per id-bearing tool call the runtime reports (Claude Code today), up to 1,024 per iteration, a child of that iteration's agent span, named execute_tool <tool name>. toolSpanTracker (internal/cli/tool_spans.go) starts the span when it handles the ToolUseEvent and ends it when it handles the matching ToolResultEvent. iterationEventHandler runs the console renderer first, then the tracker, then the Level 3 collector, so both timestamps are runner-side receipt instants on one clock, each trailing the sandbox by the pipe latency and the parser's decode of the line: the start also trails receipt by the renderer's output for the call (the renderer prints nothing for a result), and neither waits on the collector's redaction pass over its own event (events are handled one at a time, so with several calls open an instant can still trail the collector's pass over an earlier event, such as another call's large result) — the start is arguments-complete, not execution start. Attributes: gen_ai.operation.name=execute_tool, gen_ai.tool.name, gen_ai.tool.call.id; a result flagged is_error sets error.type=tool_error and status Error. A result whose stream line exceeded the parser's 1 MiB bound arrives as ToolResultEvent{Oversized} (the parser salvages the call id from the line's retained prefix) and ends its span at receipt marked fullsend.tool.result_oversized, status Unset and no error.type, since is_error was never decoded. A call still open when the stream ends — the runtime was stopped, or an over-long result line showed no id in that prefix — is closed as error.type=unanswered by Finish(), which runAgent calls once rt.Run returns, before content assembly; a call superseded by a second tool_use with the same id is ended the same way at the reuse; a result with no matching call (its tool_use line was skipped) becomes a near-zero-duration span marked fullsend.tool.unmatched=true. Events without an id — pi and codex emit none — produce no span, so the child count can be below fullsend.tool_calls, which counts every reported call, id or not; a server_tool_use block on an assistant line produces no event at all (its result never arrives as a tool_result), so it appears in neither count. The name passes through security.OutputPipeline() — Unicode normalization, then secret redaction, the same pipeline as span content — and is bounded to 256 bytes before it becomes the attribute; the span name keeps at most 128 bytes of it. The call id is scanned through the same pipeline and dropped from the span on any finding — never substituted, since a masked id could collide with another call's — while the raw bounded id still keys the open-call map, so correlation is unaffected. The tracker records at most maxToolSpansPerIteration (1,024) spans per iteration and reports the overflow, which runAgent records as fullsend.tool_spans.dropped on the agent span — a burst of agent-controlled calls must not fill the OTLP batch queue and evict the agent span, which ends after Finish() and content assembly. These spans are metadata: they carry no tool content and are emitted whether or not the Level 3 gate is on.

Level 3 content on agent spans

When the content-capture gate is on (telemetry.ContentCaptureEnabled()), runAgent constructs one contentCollector per iteration — iteration and agent span are 1:1, so a run-scoped collector would repeat earlier iterations' content on later spans — and tees the runtime's normalized event stream to it through RunParams.OnEvent.

The tee trap: supplying any OnEvent replaces the runtime's default console renderer (internal/runtime/claude.go), so the handler built by iterationEventHandler always calls the renderer first, then the tool-span tracker, then the collector — the tracker stamps span instants when it handles an event, so it must not wait on the collector's redaction pass over that event. The handler is always set — tool spans are emitted with the gate off — and with the gate off the collector is nil and inert, so console output stays byte-identical to the default renderer path.

The collector (internal/cli/content_collector.go) coalesces contiguous text/reasoning deltas, maps tool use to tool_call parts and tool results to tool_call_response parts (only the Claude parser emits ToolResultEvent and call ids today — pi and codex emit neither, #7414; the schema's required result field is response), redacts every part through security.OutputPipeline() at assembly (redaction runs before the size budget — truncating first could split a secret past recognition), enforces a 256 KiB ordered-suffix budget (the ending survives — the final answer is what consumers judge) plus an 8 KiB per-tool-result bound (tail-kept, redacted before the cut, the part marked fullsend.truncated), with exact dropped-byte accounting across content, tool names, summaries, responses, and part ids, then holds the marshaled string to maxEncodedContentBytes (255,000 — just under the one size the pilot backend is proven to accept) by trimming the oldest content again, measured on the encoding itself and still charged in raw bytes (a separate, earlier boundary — the parser's 1 MiB stream-line cap — skips oversized lines; an oversized tool_result line still yields an empty part marked fullsend.truncated). None of the three content bounds is a measured backend limit; the constants' comments and Size limits name what blocks raising them. The collector emits gen_ai.output.messages JSON following the GenAI output-messages schema, including the schema-required finish_reason from the iteration outcome. attachContent records the content and its marker attributes on the span before either finalizeAgentSpan path can end it, so failed iterations keep their content.

Consumer contract (for eval scorers and other readers of run-telemetry.jsonl): parse the gen_ai.output.messages attribute as JSON; check fullsend.content.truncated / fullsend.content.dropped_bytes before treating content as complete; masked secrets appear as the redactor's mask tokens and are counted in fullsend.content.redactions. The attribute names and shapes above are the consumption contract — see the Tracing reference.

Trace identity and TRACEPARENT propagation

resolveTraceIdentity() handles W3C trace context propagation in three steps:

Extracts TRACEPARENT and TRACESTATE from env via the W3C propagator.
Starts the root span (Consumer if remote parent, Internal otherwise).
Computes the propagated traceparent with flag preservation: if the inbound parent was valid, remote, and unsampled, the outbound traceparent keeps the unsampled flag instead of the local AlwaysSample flag. This prevents child runs from re-advertising as sampled when the parent trace opted out.

The resulting TRACEPARENT string is passed to pre-scripts, post-scripts, and the sandbox environment via childScriptEnv(). That function strips any inherited TRACEPARENT from os.Environ() and runner_env before appending fullsend's own value (issue #2779).

File exporter output format

fileExporter writes OTLP JSON with hex-encoded trace/span IDs (per the OTLP JSON spec, not base64). Each line is a complete TracesData message. Non-finite floats (NaN, Infinity) are encoded as proto3 JSON strings per the protobuf spec.

buildResourceSpans() groups SDK spans by resource and instrumentation scope, preserving insertion order. This matches the structure a backend receives via OTLP/HTTP.

Prerequisites

A local clone of the fullsend repo
Go toolchain (see go.mod for the minimum version)
Podman or Docker, if testing with a local tracing backend

How to add a new span attribute

Add the attribute.String / attribute.Int / attribute.Float64 call in run.go at the appropriate point. Use start attributes for values known when the span opens; use end attributes (set before span.End()) for values computed during the span.
Choose the attribute key namespace:
- gen_ai.* for attributes defined by the OTel GenAI semantic conventions.
- fullsend.* for project-specific attributes.
Update the attribute tables in the Tracing Reference.
No exporter changes are needed; both exporters pick up new attributes automatically.

How to test

Unit tests

go test ./internal/telemetry/...

Tests use t.Setenv for OTEL env vars and t.TempDir for the file exporter. No external backend is required.

Test seam for the OTLP exporter

newOTLPExporter is a package-level var that tests override to spy on or stub out exporter creation:

orig := newOTLPExporter
defer func() { newOTLPExporter = orig }()
newOTLPExporter = func(_ context.Context) (sdktrace.SpanExporter, error) {
    // spy or stub
}

Testing with a local backend

Start a Jaeger instance and point the exporter at it:

podman run -d --name jaeger \
  -p 16686:16686 \
  -p 4318:4318 \
  jaegertracing/jaeger

export OTEL_EXPORTER_OTLP_ENDPOINT="http://localhost:4318"
go run ./cmd/fullsend run triage ...

See Running Agents Locally for full flags. View traces at http://localhost:16686.

Replaying collected traces

hack/upload-traces.sh replays run-telemetry.jsonl files into any OTLP backend using otelcol-contrib. This is useful for inspecting traces from CI runs or from another machine without re-running the agent.

# Replay a single file
hack/upload-traces.sh run/agent-run-123/run-telemetry.jsonl \
  --endpoint http://localhost:4318

# Replay all .jsonl files under a directory
hack/upload-traces.sh run/ --endpoint http://localhost:4318

The collector runs continuously and watches for new data; press Ctrl+C when done.

Prerequisites: otelcol-contrib >= 0.120.0 on PATH (releases).

Authentication headers: otelcol-contrib ignores the standard OTEL_EXPORTER_OTLP_HEADERS env var. Edit hack/upload-traces-otelcol-config.yaml and uncomment the headers: block to set auth tokens or routing headers (e.g. x-mlflow-experiment-id).

Compatible backends

Any OTLP/HTTP-capable backend works. LLM-aware backends recognize the gen_ai.* attributes and surface GenAI dashboards (token cost rollups, prompt/completion inspection, agent-specific views) without CLI-side changes.

Backend	Local quickstart	UI
Jaeger	`podman run -p 16686:16686 -p 4318:4318 jaegertracing/jaeger`	`localhost:16686`
Arize Phoenix	`podman run -p 6006:6006 -p 4318:4318 arizephoenix/phoenix`	`localhost:6006`
MLflow >= 3.6	`uvx mlflow server`	`localhost:5000`
otel-gui	`podman run -p 4318:4318 ghcr.io/metafab/otel-gui:latest`	`localhost:4318`