Embabel Agent Observability

Unified observability for Embabel AI Agents — Automatic tracing, metrics, and LLM call integration with zero code changes.

See It In Action

Langfuse

LangSmith

Zipkin

Quick Start

Note: This library is published to the Embabel snapshot repository. Add the following repository to your pom.xml:

<repositories>
    <repository>
        <id>embabel-snapshots</id>
        <url>https://repo.embabel.com/snapshots</url>
        <snapshots>
            <enabled>true</enabled>
        </snapshots>
    </repository>
</repositories>

1. Add the core dependency

<dependency>
    <groupId>com.embabel.agent</groupId>
    <artifactId>embabel-agent-starter-observability</artifactId>
    <version>${embabel-agent.version}</version>
</dependency>

2. Add common configuration

# Embabel Observability
embabel:
  agent:
    platform:
      observability:
        enabled: true
        service-name: my-agent-app
        max-attribute-length: 4000

# Spring Boot Tracing (required)
management:
  tracing:
    export:
      enabled: true
    sampling:
      probability: 1.0  # 1.0 = 100%, 0.5 = 50%, etc. Platform default is 1.0

management.tracing.enabled was removed in Spring Boot 4; use management.tracing.export.enabled.

3. Choose your exporter

<dependency>
    <groupId>com.quantpulsar</groupId>
    <artifactId>opentelemetry-exporter-embabel</artifactId>
    <version>0.6.0</version>
</dependency>

management:
  langfuse:
    enabled: true
    endpoint: https://cloud.langfuse.com/api/public/otel  # self-hosted: http://localhost:3000/api/public/otel
    public-key: pk-lf-...
    secret-key: sk-lf-...

management:
  langsmith:
    enabled: true
    endpoint: https://eu.api.smith.langchain.com/otel  # US: https://api.smith.langchain.com/otel — /v1/traces is appended automatically
    api-key: lsv2_...
    project: my-project
    # embabel-only: true   # optional — export only Embabel/GenAI spans

<dependency>
    <groupId>io.opentelemetry</groupId>
    <artifactId>opentelemetry-exporter-zipkin</artifactId>
</dependency>

management:
  zipkin:
    tracing:
      endpoint: http://localhost:9411/api/v2/spans

docker run -d -p 9411:9411 openzipkin/zipkin

<dependency>
    <groupId>io.micrometer</groupId>
    <artifactId>micrometer-registry-prometheus</artifactId>
</dependency>

management:
  endpoints:
    web:
      exposure:
        include: prometheus, health, metrics
  prometheus:
    metrics:
      export:
        enabled: true

docker run -d -p 9090:9090 prom/prometheus
docker run -d -p 3000:3000 grafana/grafana

<dependency>
    <groupId>io.opentelemetry</groupId>
    <artifactId>opentelemetry-exporter-otlp</artifactId>
</dependency>

management:
  otlp:
    tracing:
      endpoint: http://localhost:4317

4. Done!

Your agents are now fully traced. No code changes required.

Features

Implemented

Coming Soon

Supported Backends

Tip: You can use multiple exporters simultaneously (e.g., Langfuse for traces + Prometheus for metrics).

Configuration

⚠️ Migrating from an earlier version — breaking changes

• Config prefix renamed: embabel.observability.* → embabel.agent.platform.observability.*. The old prefix is silently ignored (no startup error), so existing config stops taking effect until you rename it. Update every property in your application.yml/application.properties.
• trace-http-details now defaults to false (was true). HTTP request/response bodies, headers and params are no longer captured unless you opt in with trace-http-details: true. This avoids capturing potentially sensitive HTTP payloads by default.
• Removed properties: tracer-name and tracer-version no longer exist.
• Langfuse exporter replaced: com.quantpulsar:opentelemetry-exporter-langfuse:0.4.0 → the unified com.quantpulsar:opentelemetry-exporter-embabel:0.6.0, which exports to both Langfuse (management.langfuse.*) and LangSmith (management.langsmith.*). Update the dependency.

Tool and tool-loop spans

Tool calls are traced by the embabel.tool span (rich: name, group, correlation id, status, duration, arguments/result, error) emitted by this module's event listener, gated by trace-tool-calls. Spring AI's thinner native tool call observation is therefore always suppressed to avoid a duplicate — by an ObservationPredicate registered in this module.

The embabel.tool_loop span is opened directly at the work site in the agent core (so its hierarchy is always correct, even under parallelism). The core never reads the trace-* flags — that would couple it to this module. Instead, trace-tool-loop=false makes the same predicate drop the embabel.tool_loop span by matching its context type (ToolLoopObservationContext) — not its name: the core opens scoped spans with a placeholder name and the semantic name (embabel.tool_loop) is only applied at start(), after the predicate has run, so the context type is what the predicate can reliably match. A suppressed span becomes a no-op, so its children simply re-parent to the next live ancestor span. (Spring AI's tool call span, by contrast, carries its real name at creation and is dropped by name.)

embabel:
  agent:
    platform:
      observability:
        trace-tool-calls: false   # drop the 'embabel.tool' spans
        trace-tool-loop: false    # drop the 'embabel.tool_loop' spans

Session & user grouping

The embabel.agent span carries two ids that backends such as Langfuse use to group traces:

• gen_ai.conversation.id — the session id, taken from the last Conversation on the blackboard, falling back to the run id. Every turn of a conversation shares it.
• user.id — the per-user id, resolved from the process identity ProcessOptions.identities.forUser (the same source as OperationContext.user()), falling back to the last User bound on the blackboard. Omitted when no user is known.

How It Works

Tracing uses direct Micrometer instrumentation: the long-scoped spans (agent turn, action, LLM call, tool loop) are opened and closed with observe{} right at the work site in the agent core, on the thread doing the work. This makes the span hierarchy correct by construction — including heavy parallelism (PARALLEL mode, sub-agents, async user code) — because nesting comes from Micrometer's current-observation mechanism and cross-thread context propagation, not from reconstructing relationships out of a decoupled event stream.

Short-lived point events (LLM/embedding invocations, tool calls, planning, replan, tool-loop completion, RAG, ranking, state transitions, lifecycle, goal, dynamic agent creation) hold no scope, so they are emitted as instantaneous spans by a small event-driven listener (EmbabelSpanEventListener), nested under the current observation — or as root spans when there is none (standalone embeddings made outside an agent process, and ranking).

┌──────────────────────────────────────────────────────────────┐
│                       EMBABEL AGENT CORE                       │
│   observe{} at the work site (thin context + span name only)   │
│   embabel.agent → embabel.action → embabel.llm                 │
│                 → embabel.tool_loop                            │
└───────────────┬────────────────────────────────┬──────────────┘
                │ (long-scoped spans)             │ (events)
                │                                 ▼
                │                   ┌─────────────────────────────┐
                │                   │  EmbabelSpanEventListener    │  point spans:
                │                   │  (this module)               │  embabel.llm.invocation,
                │                   └──────────────┬──────────────┘  embabel.embedding, embabel.tool,
                │                                  │                 planning/replan/rag/ranking/
                ▼                                  ▼                 state/lifecycle/goal/dynamic
        ┌───────────────────────────────────────────────────┐
        │            ObservationRegistry                     │
        │  + Embabel span conventions (attributes)           │
        │  + tier-filter ObservationPredicate (trace-* off)  │
        └───────────────────┬───────────────────────────────┘
                            │
            ┌───────────────┴────────────────┐
            ▼                                ▼
  ┌──────────────────────┐        ┌────────────────────────┐
  │ DefaultTracing       │        │ EmbabelMetricsEvent     │  (event-driven,
  │ ObservationHandler   │        │ Listener → MeterRegistry │   independent of tracing)
  │ → OpenTelemetry      │        └────────────────────────┘
  └─────────┬────────────┘
            │
   ┌────────┼──────────┬──────────┐
   ▼        ▼          ▼          ▼
┌────────┐┌────────┐┌────────┐┌────────┐
│Langfuse││ Zipkin ││  OTLP  ││ Custom │
└────────┘└────────┘└────────┘└────────┘

Key Points:

• Long-scoped spans are instrumented directly in the core; the standard Micrometer DefaultTracingObservationHandler turns them into OpenTelemetry spans (no custom handler).
• Span attributes live in this module as GlobalObservationConventions; the core carries only thin context wrappers + span names, so it has no dependency on this module and runs without it.
• Point events become instantaneous spans via EmbabelSpanEventListener — children of the current observation, or root spans when there is none (standalone embeddings, ranking).
• Correct parent-child hierarchy even under parallelism, via context propagation — no event-stream reconstruction, no held scopes between events.
• Business metrics are event-driven and independent of tracing (metrics-enabled).
• Zero code instrumentation required; multiple exporters can run simultaneously.
• OpenTelemetry GenAI semantic conventions (gen_ai.*) for interoperability with LLM observability platforms.

Trace Hierarchy Example

Every span carries two names: the contextual name is the human-friendly label shown in the trace backend (kebab-cased by the standard handler), and the low-cardinality meter name (embabel.*, shown in parentheses below) is the stable identifier used for metrics. Below, the label is the contextual name and the parenthesised value is the meter name.

customer-service-agent                 (embabel.agent — one run() turn)
├── planning                           (embabel.planning) [goal=RequestProcessed, action_count=3]
├── analyze-request                    (embabel.action)
│   └── gpt-4o                         (embabel.llm — one LLM interaction)
│       └── tool-loop                  (embabel.tool_loop)
│           ├── gpt-4o                 (embabel.llm.invocation) [usage.input_tokens=…, output_tokens=…, cost=…]
│           ├── search-knowledge-base  (embabel.tool) [status=success]
│           └── tool-loop-completed    (embabel.tool_loop.completed) [total_iterations=2, replan_requested=false]
├── generate-response                  (embabel.action)
│   └── gpt-4o                         (embabel.llm)
├── request-processed                  (embabel.goal)
└── completed                          (embabel.lifecycle)

embabel.ranking (agent routing) is a platform-level decision with no enclosing agent process, so it is emitted as its own root span rather than nested in a turn.

Input / Output

The embabel.agent, embabel.action and embabel.tool_loop spans carry the step's input and output under the OpenInference keys input.value and output.value — which LLM-observability backends (Langfuse, etc.) render in their dedicated input/output panels:

• agent — input: the bound UserInput(s); output: the run's last result.
• action — input: the action's declared inputs read from the blackboard; output: the action's result.
• tool_loop — input: the prompt messages; output: the loop result.

Both are truncated to embabel.agent.platform.observability.max-attribute-length. These keys are vendor-neutral (OpenInference), not Langfuse-specific.

For Spring AI ChatModel spans, the ChatModelObservationFilter additionally emits the structured OTel GenAI message attributes gen_ai.input.messages / gen_ai.output.messages (JSON arrays of {role, parts:[{type, content}]}, with finish_reason on output), alongside the OpenInference input.value / output.value bridge. All payload content — these GenAI message attributes, the input.value / output.value / *.result bodies on the agent/action/tool_loop spans, and the point-span bodies (tool-call args/result, RAG query, planning/goal world state, replan reason, @Tracked args/return) — is governed by embabel.agent.platform.observability.capture-message-content (default true); set it to false to keep model/token/identity metadata while omitting every message body (the GenAI convention recommends content capture be opt-in, as it may contain PII).

Sub-agent Hierarchy

A sub-agent runs its own run() turn, so its embabel.agent span nests under the parent action that spawned it (cross-thread parent propagation handles async spawning):

orchestrator-agent                     (embabel.agent — root turn)
├── delegate-to-specialist             (embabel.action)
│   └── specialist-agent               (embabel.agent — sub-agent turn)
│       ├── specialized-task           (embabel.action)
│       │   └── claude-3-5-sonnet      (embabel.llm)
│       └── embabel.lifecycle          [state=COMPLETED]
└── embabel.lifecycle                  [state=COMPLETED]

Custom Operation Tracking with @Tracked

For tracking custom operations in your agent code, use the @Tracked annotation. It automatically creates observability spans capturing inputs, outputs, duration, and errors.

Basic Usage

@Tracked("enrichCustomer")
public Customer enrich(Customer input) {
    // Your logic here
}

With Type and Description

@Tracked(value = "callPaymentApi", type = TrackType.EXTERNAL_CALL, description = "Payment gateway call")
public PaymentResult processPayment(Order order) {
    // ...
}

Available Track Types

What Gets Captured

• Operation name (from value or method name)
• Method arguments with parameter names (e.g., {query=hello, limit=10}, truncated to max-attribute-length)
• Return value (truncated to 256 chars)
• Duration (automatic)
• Errors (automatic, with stack trace)
• Agent context (runId, agent name — when inside an agent process)

Note: Parameter names are automatically resolved via the method signature. If parameter names are not available (e.g., compiled without -parameters flag and no debug info), the output falls back to array format: [hello, 10].

Trace Hierarchy

When @Tracked methods are called within an agent execution, spans are automatically nested under the current action or agent span:

customer-service-agent              (embabel.agent)
├── process-order                   (embabel.action)
│   ├── enrich-customer             (@Tracked, PROCESSING)
│   ├── gpt-4o                      (embabel.llm)
│   └── call-payment-api            (@Tracked, EXTERNAL_CALL)
└── completed                       (embabel.lifecycle)

Important: Spring AOP Proxy Limitation

@Tracked uses Spring AOP, which is proxy-based. This means internal method calls within the same class are not intercepted:

@Component
public class MyService {

    @Tracked("step1")
    public String step1() { return "ok"; }

    public void process() {
        step1(); // this.step1() — bypasses the proxy, @Tracked NOT triggered!
    }
}

Workarounds (from simplest to most complete):

1. Extract to a separate bean (recommended):

@Component
public class MyService {
    private final MyTrackedOps ops; // injected by Spring

    public void process() {
        ops.step1(); // goes through the proxy — @Tracked works!
    }
}

@Component
public class MyTrackedOps {
    @Tracked("step1")
    public String step1() { return "ok"; }
}

2. Self-injection:

@Component
public class MyService {
    @Autowired
    private MyService self; // Spring injects the proxy, not this

    public void process() {
        self.step1(); // goes through the proxy — @Tracked works!
    }

    @Tracked("step1")
    public String step1() { return "ok"; }
}

MDC Propagation for Log Correlation

Embabel Agent context is automatically propagated into SLF4J MDC, making it easy to filter and correlate application logs by agent run or action.

MDC Keys

Logback Pattern Example

<pattern>%d{HH:mm:ss.SSS} [%thread] %-5level %logger{36} [runId=%X{embabel.agent.run_id} agent=%X{embabel.agent.name} action=%X{embabel.action.name}] - %msg%n</pattern>

This produces logs like:

14:23:45.123 [main] INFO  c.e.MyService [runId=abc-123 agent=CustomerServiceAgent action=AnalyzeRequest] - Processing request

The keys are propagated across the agent's internal thread hops (planning loop, tool loop, async fan-out), so log lines emitted off the calling thread keep the same runId/agent/action correlation. This works whether or not tracing is enabled.

To disable MDC propagation:

embabel:
  agent:
    platform:
      observability:
        mdc-propagation: false

Business Metrics (Micrometer)

When a MeterRegistry is available (e.g. via micrometer-registry-prometheus), the module automatically registers the following business metrics, independent of the tracing implementation chosen:

These metrics are exploitable by Prometheus and Grafana out of the box. To disable:

embabel:
  agent:
    platform:
      observability:
        metrics-enabled: false

Roadmap

Requirements

• Java 21+
• Spring Boot 3.5+
• Embabel Agent 0.5.0+

License

Apache License 2.0 - See LICENSE for details.

Contributing

Contributions are welcome! You can help by:

• Reporting bugs or suggesting features
• Submitting pull requests
• Adding or improving tests