Observability

Last modified: 2026-07-09

SBproxy ships metrics, logs, and traces from one process. This guide covers the Wave 1 substrate: the SLO catalog, the metric label budget, the log schema and redaction policy, the trace propagation contract, the health endpoints, the dashboards, and the reference Compose stack you can boot in one command.

Three pillars

Pillar	Surface	Default state	Where it goes
Metrics	/metrics (Prometheus / OpenMetrics)	Always on	Prometheus, scraped on a 15 s cadence
Logs	stdout and configurable sinks	Always on, JSON-line	Loki, S3, customer collectors
Traces	OTLP exporter	Off by default; opt in per deployment	Tempo, Jaeger via the OTel Collector

All three speak the same correlation triple: every log line and every span attribute carries request_id (UUIDv7 rendered as 32 lowercase hex chars without hyphens; RFC 9562 monotonic + time-ordered), trace_id (32-hex), and span_id (16-hex). One inbound 402 with one trace stitches metrics, logs, and traces together without join-by-timestamp. The UUIDv7 leading 48 bits are a Unix-millisecond timestamp so a ClickHouse ORDER BY request_id partitions naturally by ingest time.

The correlation_id policy threads one identifier through logs, webhooks, and the upstream (config).

Configuration

The currently shipped schema lives under proxy.observability: and groups the log (tracing-subscriber filter + format + sampling) and telemetry (OTLP exporter) blocks. When the block is absent, CLI flags and env vars are the only source of truth.

proxy:
  observability:
    log:
      level: info                  # debug | info | warn | error
      format: compact              # compact | pretty | json
      sampling:
        info: 1.0                  # fraction of info lines kept
        debug: 0.1
        trace: 0.01
    telemetry:
      enabled: true
      endpoint: "http://otel-collector:4317"
      transport: grpc              # grpc | http
      service_name: "sbproxy"
      sample_rate: 0.1             # head ratio for unsampled roots
      always_sample_errors: true   # 100% on 5xx / policy block paths
      keep_over_budget_usd: 1.00   # keep completed traces at/above this cost
      keep_slower_than_secs: 2.0   # keep completed traces at/above this latency
      propagation: w3c             # w3c | b3 | jaeger
      resource_attrs:
        deployment.environment: "prod"
        service.version: "${SBPROXY_VERSION}"
      export_metrics: false        # mirror metrics over OTLP
      metrics_interval_secs: 30

sample_rate controls normal traffic with parent-based trace-id ratio sampling. Inbound sampled W3C parents are kept. Locally dropped spans are still recorded until completion so always_sample_errors, keep_over_budget_usd, and keep_slower_than_secs can export the traces operators usually need most.

Sinks

The observability.log.sinks: block fans every emitted structured-log record out to one or more declared sinks. Each sink picks its own destination (stdout, stderr, rotating file, OTLP collector), wire format, and redaction profile. When no sinks are declared the legacy single tracing subscriber drives stdout exactly as it did before; the fan-out path only lights up once the operator declares at least one sink.

proxy:
  observability:
    log:
      sinks:
        - name: stdout
          target: access_log
          format: json
          output: { type: stdout }
          profile: internal
        - name: stderr-audit
          target: audit_log
          format: json
          output: { type: stderr }
        - name: file-archive
          target: audit_log
          format: json
          output:
            type: file
            path: /var/log/sbproxy/audit.json
            max_size_mb: 100
            max_backups: 7
            compress: true
          profile: internal
        - name: otel-collector
          target: access_log
          format: json
          output:
            type: otlp
            endpoint: http://otel-collector:4318/v1/logs
            transport: http
            timeout_secs: 5
          profile: external

Field schema:

• name is unique within the declaring scope. Duplicates within a scope are warn-logged today and reserved for a hard reject in a follow-up patch.
• target selects the internal channel: access_log | error_log | audit_log | trace_exporter | external_log. A sink only sees records emitted on the channel it subscribes to.
• format overrides the parent proxy.observability.log.format for this sink. Today every variant emits one JSON object per line; pretty re-renders with indentation.
• output is the where: see the four output types below.
• profile is the redaction shape: internal keeps JA3/JA4 fingerprints and raw query strings; external strips them. Proxy-scope sinks default to internal; tenant- and origin-scope sinks default to external because the downstream backend is usually outside the operator's trust boundary.

Output types

type	Fields	Notes
stdout	(none)	Locks the process stdout per write. Default for a freshly-installed proxy.
stderr	(none)	Useful for routing the audit channel separately from access on systemd-journald.
file	path, max_size_mb, max_backups, compress	Reuses the access-log rotation + gzip stack. Defaults: 100 MiB rotation, 7 backups, gzip on.
otlp	endpoint, transport, timeout_secs	Wraps opentelemetry_otlp::LogExporter behind a batch processor. Inherits service_name, resource_attrs, and (when omitted) transport from the top-level telemetry: block.

Sink scopes

Sinks can be declared at three scopes, each with a different filter:

• proxy.observability.log.sinks: (proxy scope) receives every record. This is where general-purpose stdout / file / OTLP sinks live.
• tenants[].observability.log.sinks: (tenant scope) receives only records whose resolved Principal.tenant_id matches the tenant id. Cross-tenant records never reach a tenant-scoped sink.
• origins[].observability.log.sinks: (origin scope) receives only records whose stamped route matches the origin's hostname. Useful for an origin that ships its logs to a tenant-specific Loki instance.

A worked example with two tenants:

proxy:
  tenants:
    - id: acme
      observability:
        log:
          sinks:
            - name: acme-loki
              target: access_log
              output:
                type: otlp
                endpoint: http://loki-acme:4318/v1/logs
                transport: http
    - id: beta
      observability:
        log:
          sinks:
            - name: beta-stdout
              target: access_log
              output: { type: stdout }
              profile: external

A record emitted with tenant_id = Some("acme") reaches only acme-loki; a record with tenant_id = Some("beta") reaches only beta-stdout; a record without a tenant id reaches neither tenant sink but still reaches any proxy-scope sinks.

OTLP-logs exporter

The otlp output ships each line through an OpenTelemetry BatchLogProcessor to the configured collector. Every record stamps the OTel resource attributes service.name = sbproxy (or the operator's override), service.version = <crate version>, and service.instance.id = <hostname>; any telemetry.resource_attrs: entries layer on top.

The level-to-severity mapping follows the OTel spec:

Structured-log level	OTel SeverityNumber
trace	1
debug	5
info	9
warn	13
error, fatal	17

A reference Collector pipeline that accepts these logs and forwards them on to Loki:

receivers:
  otlp:
    protocols:
      http:
        endpoint: 0.0.0.0:4318
      grpc:
        endpoint: 0.0.0.0:4317

processors:
  batch:
    timeout: 5s
    send_batch_size: 1024

exporters:
  # Loki 3.x accepts native OTLP at /otlp; no Promtail or Fluent Bit needed.
  otlphttp/loki:
    endpoint: http://loki:3100/otlp
    tls:
      insecure: true

service:
  pipelines:
    logs:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlphttp/loki]

Operators that already run an OTel Collector for traces can add the logs pipeline above and point the proxy's OTLP-logs sink at the same endpoint. The batch processor in the sink keeps the proxy's hot path non-blocking; flushes happen on SIGHUP and on shutdown.

Metrics

Naming and labels

Every metric name starts with sbproxy_. The label set is closed: a label that is not on the budget table below is a CI failure. The closed set protects the scrape from cardinality blow-ups when an attacker rolls a fresh UA per request.

The Wave 1 substrate adds five labels: agent_id, agent_class, agent_vendor, payment_rail, content_shape. agent_id, agent_class, and agent_vendor are bounded to the agent-class registry plus three reserved sentinels (human, unknown, anonymous); payment_rail and content_shape are closed enums.

SLO catalog

ID	Pillar	SLI	Target	Window	Tier on breach
SLO-AVAIL-INBOUND	Substrate	inbound request availability (non-5xx / total)	99.9%	30d	Page
SLO-LATENCY-P95	Substrate	inbound p95 latency excl. rail wait	< 30 ms	5 min sustained	Ticket
SLO-LATENCY-P99	Substrate	inbound p99 latency excl. rail wait	< 50 ms	5 min sustained	Page
SLO-LEDGER-REDEEM	Ledger	redeem success rate	99.95%	30d	Page
SLO-LEDGER-LATENCY	Ledger	redeem p99 latency	< 200 ms	5 min sustained	Ticket
SLO-RAIL-SETTLE	Rails (per rail)	settle success rate	99.5%	7d	Page
SLO-RAIL-QUORUM	Rails	facilitator quorum (>= 1 healthy per chain)	100%	instant	Page (immediate)
SLO-AUDIT-WRITE	Audit	batch-write success	100%	24h	Page (immediate)
SLO-AUDIT-LATENCY	Audit	emit-to-durable latency p99	< 5 s	1h sustained	Ticket
SLO-DR-RESTORE	DR	restore drill	succeed monthly	calendar	Page on missed
SLO-WEBHOOK-IN	Webhooks (in)	inbound verification success	99.9%	7d	Ticket
SLO-WEBHOOK-OUT	Webhooks (out)	outbound delivery success (incl. retries)	99%	7d	Ticket
SLO-CONFIG-RELOAD	Config	hot-reload success	100%	24h	Page
SLO-BOT-AUTH-DIR	Bot Auth	directory freshness (TTL not exceeded)	99.9%	7d	Ticket
SLO-CARD-BUDGET	Substrate	per-metric series count under cap	100%	continuous	Log-only (CI gate)

PromQL recording rules pre-compute each SLI at 1m, 5m, 1h, 6h, and 24h windows. Burn-rate alerts use the multi-window pattern from the SRE workbook (5m AND 1h at 14.4x for page tier, 30m AND 6h at 6x, 2h AND 24h at 3x for ticket). The full rule set lives in deploy/alerts/.

Cardinality budget

Metric family	Cardinality cap	Notes
sbproxy_requests_total	50 000	Labels: hostname, method, status, agent_id, agent_class, agent_vendor, payment_rail, content_shape. agent_id is the sanitized registry id, never a raw UA-derived value.
sbproxy_request_duration_seconds_bucket	100 000	Labelled by hostname, plus buckets.
sbproxy_policy_triggers_total	20 000	Labels: origin, policy_type, action, agent_id, agent_class.
sbproxy_ledger_redeem_duration_seconds_bucket	10 000	Labels: host, outcome, plus buckets. There is no separate _total counter; derive counts from the histogram's _count series.
sbproxy_outbound_request_duration_seconds_bucket	30 000	Labels: host, method, status, plus buckets. There is no separate _total counter.
sbproxy_audit_emit_duration_seconds_bucket	5 000	Labels: channel, outcome, plus buckets. There is no separate _total counter.
sbproxy_script_compile_total	12	Labels: engine (cel|lua|js|wasm), result (ok|parse_error|sandbox_reject).
sbproxy_script_invocations_total	20	Same engine, plus result (ok|runtime_error|timeout|memory_cap|instruction_cap).
sbproxy_script_duration_seconds_bucket	52	engine label only; histogram buckets 0.1ms..10s.
sbproxy_script_reloads_total	12	Same labels as compile; counts hot-reload events separately so reload churn surfaces independently.
sbproxy_rate_limit_decisions_total	4 000	Labels: policy (sanitised route pattern), result (allow|throttle_route|throttle_tenant|disabled).
sbproxy_idempotency_cache_results_total	16	Labels: backend (default), result (hit|miss|conflict|not_applicable).
sbproxy_idempotency_cache_duration_seconds_bucket	11	backend label only; histogram buckets 50us..1s.
sbproxy_response_body_bytes_bucket	18	Labels: direction (pre_compress|post_compress); histogram buckets 256B..16MiB.
sbproxy_compression_decisions_total	16	Labels: codec (gzip|br|zstd|identity), result (applied|skipped_size|skipped_accept|disabled).
sbproxy_compression_ratio_bucket	40	Labels: codec; histogram of post/pre size when compression applied.
sbproxy_plugin_registered_total	500	Labels: kind (action|auth|policy|transform|enricher), plugin (sanitised). Emitted once at startup per registration.
sbproxy_plugin_init_total	1 500	Labels: kind, plugin, result (ok|config_invalid|panic).
sbproxy_plugin_init_duration_seconds_bucket	18 000	Same labels as _init_total plus 12 histogram buckets 100us..10s.
sbproxy_acme_renewals_total	6	Labels: result (ok|http_error|order_invalid|account_invalid|rate_limited|other).
sbproxy_acme_renewal_duration_seconds_bucket	60	Same result plus 10 histogram buckets 100ms..5min.
sbproxy_ocsp_fetch_total	5	Labels: result (ok|http_error|parse_error|unknown_status|no_responder).
sbproxy_cert_expiry_seconds	256	Labels: host (sanitised). Gauge; negative means already expired.
sbproxy_vault_resolution_total	200	Labels: backend (sanitised), result (ok|not_found|backend_error|denied).
sbproxy_vault_resolution_duration_seconds_bucket	2 400	Same labels plus 12 histogram buckets 100us..5s.
sbproxy_transport_requests_total	28	Labels: protocol (h1|h2|h3|grpc|grpc_web|graphql|websocket), result (ok|client_error|upstream_error|timeout).
sbproxy_transport_duration_seconds_bucket	364	Same labels plus 13 histogram buckets 100us..10s.
sbproxy_grpc_status_total	17	Labels: code (canonical lowercase name; closed enum from tonic).
sbproxy_mcp_tool_dispatch_total	4 000	Labels: tool (sanitised), result (ok|tool_error|tool_not_found|policy_denied).
sbproxy_mcp_tool_dispatch_duration_seconds_bucket	12 000	tool label plus 12 histogram buckets 100us..10s.
sbproxy_mcp_resource_fetch_total	4	Labels: result (ok|not_found|upstream_error|policy_denied).
sbproxy_mcp_federation_peers_up	1	Gauge; live federation peer count as of the last refresh.
sbproxy_operator_reconcile_total	8	Labels: kind (sbproxy|sbproxyconfig), result (ok|conflict|backend_error|crd_invalid).
sbproxy_operator_reconcile_duration_seconds_bucket	22	kind label plus 11 histogram buckets 1ms..60s.
sbproxy_operator_leader_transitions_total	3	Labels: result (elected|renewed|lost).
sbproxy_operator_leader_is_leader	1	Gauge; 1 when this replica holds the lease.
sbproxy_tokens_attributed_total	8 000	Labels: project (sanitised), user (sanitised), tag (sanitised; first element of the virtual key's tags: list with fan-out per tag), direction (input|output). Cardinality not bounded by a fixed cap; the existing sbproxy_label_cardinality_overflow_total counter fires when any label exceeds budget. Sits next to sbproxy_ai_tokens_total{hostname,provider,direction} and indexes the same observation by who-paid attribution.
sbproxy_label_cardinality_overflow_per_tenant_total	8 000	Labels: metric (sanitised name of the demoted family), label (sanitised label key that overflowed), tenant_id. Same demotion signal as sbproxy_label_cardinality_overflow_total but partitioned by tenant so a noisy-tenant root-cause investigation does not have to scan every metric.
sbproxy_a2a_hops_total	60	Labels: route, spec (a2a-spec version), decision (allow|deny|warn). Counts each per-request A2A hop the proxy observes.
sbproxy_a2a_chain_depth_bucket	60	route, spec; histogram buckets 1..32 chain hops. Tracks A2A call-graph depth before truncation.
sbproxy_a2a_denied_total	40	Labels: route, reason (depth_cap|policy_block|loop_detected|other). Per-request denial counter on the A2A surface.
sbproxy_agent_budget_decisions_total	400	Labels: agent_id (sanitised, capped via the same demotion path as other agent_*) outcome (allow|throttle|deny). Drives the per-agent budget enforcement audit.
sbproxy_agent_detect_total	3 000	Labels: agent_id (sanitised, empty when anonymous), provenance (signed|unsigned-named|unsigned-anonymous). Per-request agent-detect scorer verdicts.
sbproxy_agent_detect_score_bucket	11	Histogram buckets over the 0-100 agent-detect score. No labels.
sbproxy_agent_detect_inference_seconds_bucket	9	Histogram buckets 50us..10ms for in-process scorer latency. No labels.
sbproxy_object_authz_violations_total	200	Labels: origin, kind (bola|bfla|enumeration). Counts BOLA / BFLA / enumeration violations the object-authz policy refused.
sbproxy_waf_persistent_blocks_total	600	Labels: origin, event (rule_match|ip_blocklisted|anomaly_threshold), key_kind (ip|jwt_sub|api_key|session). Counts the WAF blocks that landed on the persistent (cross-process) blocklist as opposed to the in-process rate-limit decision path.
sbproxy_bot_auth_nonce_replay_total	50	Labels: policy (sanitised). Counts requests rejected because the Web-Bot-Auth nonce was already seen within the replay window.
sbproxy_jwks_unknown_kid_refetch_total	6	Labels: result (ok|backend_error|kid_still_missing). Counts on-demand JWKS refetches triggered by an unknown kid in a presented JWT.
sbproxy_mtls_handshake_total	5	Labels: result (ok|cert_invalid|cert_expired|no_client_cert|other). Counter on the mTLS path; pair with sbproxy_cert_expiry_seconds to alert before certs expire.
sbproxy_ocsp_staple_age_seconds	256	Labels: host (sanitised). Gauge of the age in seconds of the currently stapled OCSP response per host. Should stay well under the OCSP nextUpdate minus the renewal margin.
sbproxy_synthetic_probe_failures_total	32	Labels: reason (timeout|status_5xx|tls_handshake|connect|dns|other). Background-probe failure counter; signals an upstream gone bad before customer traffic notices.
sbproxy_capture_dropped_total	6 000	Labels: workspace (sanitised), dimension (token|cost|attribution|other), reason (queue_full|backend_error|policy_block|budget_exhausted). Per-workspace tokenomics capture-drop counter (rolls up the budget-dropped sub-counter below).
sbproxy_capture_budget_dropped_total	2 000	Labels: workspace (sanitised), dimension (token|cost|attribution|other). Subset of sbproxy_capture_dropped_total for the budget-exhausted reason; carried separately so a budget-tuning loop can isolate this signal.
sbproxy_dedup_cache_size	1	Gauge; current entry count in the in-memory dedup cache. Drives the LRU-eviction alert.
sbproxy_mirror_state_drift_total	1	Counter; per-request increments when the request-mirror's primary and shadow responses diverge enough that a downstream replay would notice. Always sample to a debug log so the trigger is investigatable.
sbproxy_outbound_webhook_attempts_total	8 000	Labels: tenant_id, event_type (sanitised), result (ok|http_4xx|http_5xx|timeout|retry_exhausted). Per-tenant outbound webhook delivery counter; pair with the SLO-WEBHOOK-OUT row above for the success-rate burn.
sbproxy_policy_audit_events_total	1 200	Labels: verdict (allow|deny|warn), surface (http|mcp|a2a|admin), policy_id (sanitised). Per-event audit-channel counter; the policy-decision path emits one per evaluated policy.
sbproxy_policy_audit_events_dropped_total	40	Labels: tenant (sanitised). Counts the policy-audit events dropped because the per-tenant queue was full. A non-zero rate here means the operator should raise policy.audit.queue_size or shed load.
sbproxy_policy_decision_duration_seconds_bucket	60	Labels: surface; histogram buckets 100us..1s. Time-to-decision per policy surface. Pair with sbproxy_policy_evaluation_duration_seconds_bucket for end-to-end policy latency.
sbproxy_mcp_policy_hook_invocations_total	2 000	Labels: verdict (allow|deny|warn), mcp_server (sanitised), tool_name (sanitised). Counts per-tool MCP policy-hook decisions.
sbproxy_judge_calls_total	60	Labels: provider (openai|anthropic|...), verdict (pass|fail|abstain), cached (true|false). Counter for the AI judge surface (rubric / scorer eval calls).
sbproxy_judge_latency_seconds_bucket	240	Labels: provider, cached; histogram buckets 100ms..30s. Per-judge call latency.
sbproxy_judge_cost_usd	10	Labels: provider. Counter; per-provider judge spend in USD.
sbproxy_judge_budget_exhausted_total	40	Labels: tenant. Counts judge calls refused because the per-tenant judge budget was exhausted.
sbproxy_ai_tokens_attributed_total	8 000	Labels: provider, model, direction (input|output), project, feature, team, agent_type, environment. The unified attribution token counter for AI traffic; same shape as the non-AI sbproxy_tokens_attributed_total but tagged with provider / model.
sbproxy_ai_cost_dollars_attributed_total	8 000	Labels: same shape as sbproxy_ai_tokens_attributed_total but valued in USD. Pair with the tokens counter to derive the per-attribution unit cost.
sbproxy_ai_wasted_tokens_total	8 000	Labels: kind (cancelled|retried|cached|guardrail_blocked|other) plus the standard attribution labels. Counts tokens spent that did NOT survive to a useful response. Drives the FOCUS waste-signal export.
sbproxy_ai_wasted_cost_dollars_total	8 000	Same shape as sbproxy_ai_wasted_tokens_total but valued in USD.
sbproxy_ai_cascade_tier_outcomes_total	200	Labels: tier (the cascade-rule tier name, sanitised), outcome (advanced|blocked|served). Counts each cascade-rule tier outcome the AI router observed.
sbproxy_ai_native_bypass_total	100	Labels: inbound_format, provider_format. Counts requests where the inbound surface format matched the provider format so the AI dispatch could bypass the translate-and-re-translate path.
sbproxy_ai_output_throughput_tokens_per_second_bucket	800	Labels: provider, model; histogram buckets 1..1000 tokens/sec. Per-completion output throughput; pair with sbproxy_ai_ttft_seconds_bucket for the full latency story.
sbproxy_ai_ratelimit_rejected_total	1 000	Labels: axis (provider|model|virtual_key), key_hash (truncated stable hash of the rate-limited key), model. Counts AI requests refused at the per-axis rate limiter before reaching the provider.
sbproxy_ai_semantic_cache_similarity_bucket	200	Labels: provider; histogram buckets 0.0..1.0 of cosine similarity between the request embedding and the cached entry. Lets the operator tune the cache-hit threshold from observed similarity distribution.
sbproxy_ai_shadow_inflight	1	Gauge; live in-flight shadow-evaluation count. Pair with sbproxy_ai_shadow_dropped_total to alert when shadow runs back up.
sbproxy_ai_shadow_dropped_total	1	Counter; shadow evaluations dropped because the queue or in-flight cap was hit.
sbproxy_ai_shadow_timeout_total	1	Counter; shadow evaluations dropped because the per-eval timeout fired.
sbproxy_ai_token_estimate_error_ratio_bucket	200	Labels: model; histogram buckets (estimate - actual) / actual between -1 and +1. Drives the pre-flight estimator's accuracy alert.

Hard rule: request_id, session_id, and user_id are never label values on Prometheus metrics; they live as span attributes (under traces) and log fields (under logs). agent_id IS a label, but only in its sanitized form: values are bounded to the agent-class registry plus the reserved sentinels, and anything outside that set demotes rather than minting a new series. Raw high-cardinality identifiers (a per-request UA string, an unregistered agent name) never become label values.

When a budget is exhausted the offending label demotes to __other__ and sbproxy_label_cardinality_overflow_total increments. The metric update still happens; a demoted bucket is preferable to a missing one because gaps look like real traffic dips.

Fleet totals across a cluster

Metrics are per-instance: each process exposes only its own counters at /metrics. The default way to see cluster-wide numbers is an external Prometheus that scrapes every instance and sums with PromQL; the bundled Grafana dashboards already do this, so a Prometheus deployment needs nothing extra here.

For deployments running the mesh key tier without a Prometheus, one node can report fleet totals directly. Each node periodically publishes a small allow-list of sbproxy_* totals into the mesh, and GET /admin/cluster/metrics returns the summed values plus the node count. This is a convenience for a single-pane view without a metrics stack, not a replacement for Prometheus: the set is curated, the cadence is coarse, and it only reports while the mesh tier is on (otherwise the endpoint returns 404). Prefer Prometheus for anything beyond an at-a-glance total.

Logs

Structured-log schema

JSON-line, UTF-8, one object per line. Field order is not significant but emitters write top-level fields in the order below for grep-ability.

Required on every line:

Field	Type	Notes
ts	string (RFC 3339 UTC, ms precision)	2026-04-30T14:23:45.123Z
level	string enum	trace, debug, info, warn, error, fatal
msg	string	Human-readable message
target	string	Module path
event_type	string enum	See list below
schema_version	string	"1" for the Wave 1 schema

Required when the line is request-scoped:

Field	Type	Notes
request_id	string (32 lowercase hex, UUIDv7)	The proxy-minted correlation id described at the top of this page. Note the RequestEvent envelope's own request_id field is a ULID, a different format minted for that stream.
trace_id	string (32 hex)	Current OTel trace id
span_id	string (16 hex)	Current OTel span id
tenant_id	string	Workspace id; default in OSS
route	string	Origin route key

Per-request lifecycle lines (request_started, request_completed, request_error) carry the same body as RequestEvent (agent_id, agent_class, rail, shape, status_code, latency_ms, error_class).

Event types pinned for Wave 1: request_started, request_completed, request_error, policy_evaluated, policy_blocked, action_challenge_issued, action_redeemed, ledger_call, audit_emit, notify_dispatch, boot, config_reload, health_status_change.

Redaction policy

Sensitive fields are matched by field key, not by value heuristics. Field names that the redactor matches: authorization, proxy-authorization, cookie, set-cookie, x-stripe-signature, stripe-signature, *_secret, *_token, *_key, prompt, messages, ja3, ja4.

Each match replaces the value with a marker. As of schema v2, every marker uses the [REDACTED:<NAME>] shape (the pre-v2 <redacted:name> form is gone):

{ "headers": { "authorization": "[REDACTED:AUTHORIZATION]" } }
{ "stripe_sk": "[REDACTED:STRIPE_SECRET_KEY]" }
{ "messages": "[REDACTED:PROMPT_BODY]" }

Operator-extensible redaction

The built-in denylist above is the security baseline and runs first. Operators add their own field-key entries and regex masks under proxy.observability.log.redact::

proxy:
  observability:
    log:
      redact:
        fields:
          - x-internal-token
          - internal_account_id
        patterns:
          - name: customer_uuid
            pattern: 'cust_[a-z0-9]{20}'
            replacement: '[REDACTED:CUSTOMER_UUID]'
          - name: internal_account
            pattern: 'acct-\d{6,12}'
            # replacement omitted: defaults to [REDACTED:INTERNAL_ACCOUNT]

• fields: is additive on the built-in baseline. Matched lowercase. Cannot disable a built-in entry.
• patterns: is a list of named regexes applied to the rendered JSON after the field-key pass. Compiled once at config load; an invalid regex is logged at warn and skipped (the rest of the block still installs). replacement: defaults to [REDACTED:<NAME_UPPER>] when omitted.

Tenant-scope and origin-scope redact additions

The fields: and patterns: blocks above also accept tenant-scope and origin-scope additions. Each scope inherits the parent and adds its own entries; patterns: additionally honours a disable: opt-out by pattern name. fields: is additive-only at every scope; a tenant or origin cannot disable a proxy-level field denylist entry because the security baseline always applies.

proxy:
  observability:
    log:
      redact:
        fields: [x-internal-token]
        patterns:
          - name: customer_uuid
            pattern: 'cust_[a-z0-9]{20}'
  tenants:
    - id: acme-corp
      observability:
        log:
          redact:
            fields: [x-acme-license]
            patterns:
              - name: acme_account
                pattern: 'acct-\d{6,12}'
            disable: [customer_uuid]   # opt out of a proxy-level rule
origins:
  "api.acme.example.com":
    tenant_id: acme-corp
    action:
      type: proxy
      url: https://acme-upstream.internal
    observability:
      log:
        redact:
          patterns:
            - name: internal_id
              pattern: '\binternal-[a-f0-9]{16}\b'
          disable: [acme_account]      # opt out of a tenant-level rule

Resolution order at emit time:

built_in_denylist
  → proxy.fields
    → tenant.fields           (inherited additive)
      → origin.fields         (inherited additive)
        → proxy.patterns
          → tenant.patterns   (proxy minus tenant.disable, then add tenant.patterns)
            → origin.patterns (parent minus origin.disable, then add origin.patterns)
              → pii.rules     (composed per the pii: block; see below)

The composition runs once per (tenant, origin) pair at config-compile so the hot path is a single HashMap lookup keyed on (record.tenant_id, record.route). Unknown rule names + invalid regexes are warn-logged with the scope label (proxy / tenant <id> / origin <hostname>) and the rest of the block still installs.

Built-in PII detector

Operators can enable the rule-driven PII detector from sbproxy-security as a fourth redaction pass. It runs after the field-key pass and the regex pass against the rendered JSON. The detector ships with built-in rules for email, US SSN, credit card (Luhn-validated), US phone, IPv4, IBAN, and common API key shapes (OpenAI, Anthropic, AWS access key, GitHub PAT, Slack token).

proxy:
  observability:
    log:
      redact:
        pii:
          enabled: true
          # rules: select a subset by name; empty means "all defaults"
          rules:
            - email
            - us_ssn
            - credit_card
          # disable: subtract from the selected set
          disable:
            - ipv4

• enabled: false (or absent) is the default; the PII pass is skipped entirely.
• rules: selects which built-in rules to install. Empty means all defaults. Unknown names are logged at warn and skipped (the install continues with the rest).
• disable: subtracts names from the resolved set. Useful when rules: is empty but you want everything except, say, ipv4.
• Default replacement is [REDACTED:<RULE_NAME_UPPER>] (e.g. [REDACTED:EMAIL]).
• The PII pass is anchor-prefilter accelerated (Aho-Corasick), so adding rules carries no measurable overhead on logs that contain none of them.

Tenant-scope PII

A tenant can author its own pii: block under tenants[].observability.log.redact.pii. The tenant-scope block composes on top of the proxy-scope block: the tenant inherits the proxy's enabled flag and its rule set, adds the tenant's rules: entries, and subtracts the tenant's disable: entries. An explicit enabled: false opts the tenant out even when proxy scope has the pass on, useful when one tenant is a regulated workload (HIPAA, PCI) that wants a stricter or laxer rule set than the rest of the fleet:

proxy:
  observability:
    log:
      redact:
        pii:
          enabled: true
          rules: [email, us_ssn]
  tenants:
    - id: hipaa-tenant
      observability:
        log:
          redact:
            pii:
              enabled: true
              rules: [email, us_ssn, hipaa_mrn, hipaa_patient_id]
              disable: [phone_us]

In this example, hipaa-tenant inherits email + us_ssn from the proxy, adds hipaa_mrn + hipaa_patient_id, and drops phone_us from the active set. Every other tenant continues to run only the proxy-scope set. A tenant id appearing here that is not declared under proxy.tenants[].id is rejected by config compile (the same rule that governs origin.tenant_id).

Origin-scope PII

An origin can author its own pii: block under origins[hostname].observability.log.redact.pii. The origin-scope block composes on top of the tenant-scope block (or the proxy-scope block when the origin has no tenant_id). The same inherit + extend + disable rules apply, one level deeper:

proxy:
  observability:
    log:
      redact:
        pii:
          enabled: true
          rules: [email, us_ssn]
  tenants:
    - id: hipaa-tenant
      observability:
        log:
          redact:
            pii:
              rules: [hipaa_mrn, hipaa_patient_id]
              disable: [phone_us]
origins:
  "api.acme.example.com":
    tenant_id: hipaa-tenant
    action:
      type: proxy
      url: https://acme-upstream.internal
    observability:
      log:
        redact:
          pii:
            rules: [billing_account]

api.acme.example.com resolves the tenant hipaa-tenant first (which itself inherits from the proxy scope), then adds billing_account on top, giving an active rule set of email + us_ssn + hipaa_mrn + hipaa_patient_id + billing_account (with phone_us still disabled, inherited from the tenant).

Resolution rules

• Resolution at emit time walks origin scope first, then the origin's tenant scope, then the proxy scope. The most-specific scope that authored a block wins on the enabled flag.
• A scope that omits enabled: inherits the parent scope's flag. A scope that sets enabled: false explicitly opts out, even when the parent enables the pass.
• The rule set inherits + extends + subtracts at each level: parent rules carry through, the child's rules: are added, the child's disable: is removed last.
• Unknown rule names at any scope are warn-logged at startup and skipped. The install continues with the rest of the resolved set so an operator typo does not silently disable the whole pass.

Reversible PII redaction (AI origins)

Customer copilots and internal assistants need the LLM to personalise its response with the same value the user typed (the customer's name, order number, or email). A destructive redactor would replace that value with [REDACTED:EMAIL] on the way out, the LLM would echo the marker back, and the response would no longer feel personal. The reversible pass solves this: the request body is masked with a placeholder before forwarding upstream, the LLM responds with the placeholder echoed in its reply, and the gateway restores the original value before writing the response to the client. The original lives only in memory for the request lifetime; it is never written to access log, audit log, or trace span.

Opt-in per rule via reversible: true on the pii: block, which sits inside the ai_proxy action (the same placement as examples/pii-redaction/sb.yml):

origins:
  "copilot.example.com":
    action:
      type: ai_proxy
      providers:
        - name: openai
          api_key: ${OPENAI_API_KEY}
      pii:
        enabled: true
        defaults: false
        redact_request: true
        rules:
          - name: email
            pattern: '\b[a-z0-9._%+-]{1,64}@[a-z0-9.-]{1,255}\.[a-z]{2,63}\b'
            reversible: true
            mask_template: "<placeholder:email:%d>"
          - name: credit_card
            pattern: '\b\d(?:[ -]?\d){12,18}\b'
            validator: luhn
            reversible: false   # never restored; PCI scope

• reversible: false (default) is the destructive behaviour described above.
• reversible: true records a (placeholder, original) pair for every match into the request context.
• mask_template: defaults to <placeholder:<rule_name>:%d>; %d is substituted with a per-request, per-rule counter starting at 0 so two matches of the same rule get distinct placeholders.
• On the response side the gateway walks the body once and replaces every recorded placeholder with the original.
• If the LLM emits a <placeholder:<rule>:N> shape that the request did not capture (model hallucination or prompt-injection probe), the placeholder is left in the response and sbproxy_ai_reversible_redaction_miss_total{rule} is incremented. The caller sees the synthetic value verbatim.

Streaming responses

The SSE streaming relay restores placeholders before each chunk is written to the client. Restoration is chunk-aware: a placeholder shape that lands across two network reads is held back at the chunk boundary until the closer arrives, then surfaced as the restored original in the next emitted chunk. The hold-back buffer is bounded at 64 bytes; a lone < that never closes (binary stream interleaved with text, or a truncated placeholder shape) is flushed verbatim once the buffer hits the cap so the stream never stalls waiting on a synthetic closer. On a clean stream end the relay flushes any final carry as-is; a malformed <placeholder:... left in the carry is emitted verbatim, with the miss counter incremented for any complete-but-uncaptured shape found in the flushed bytes.

When no reversible PII rule fires on the request the streaming path short-circuits per chunk and pays no overhead. Origins that never configure reversible rules see byte-forward streaming unchanged.

Idempotency and reversible PII

When an AI origin has both an idempotency: block and reversible PII rules, the idempotency cache stores the restored response body, not the placeholder shape. The cache key includes a hash of the request body, so a genuine hit guarantees the replay request is byte-identical and would produce the same capture map; storing the restored bytes avoids re-running restoration on every replay and keeps placeholder shapes out of the cache (which dashboards and audit replays sometimes surface). The same logic applies to the non-streaming chat-completions relay: restore runs before both the cache write and the response send.

Semantic cache co-existence

Reversible PII redaction and semantic caching cannot safely co-exist on the same origin. The semantic cache keys responses on a similarity hash of the prompt, so two requests that share a prompt shape but carry different captured originals (different customer names, different order numbers) can hash to the same cache key. A cache hit would surface the prior request's placeholders restored against the new request's capture map, which is the wrong customer's data on the wire.

The gateway resolves this at config validation: when an AI origin declares any pii.rules[].reversible: true AND a semantic_cache: block, the semantic cache is dropped from the compiled config and a warning is logged. The cache is silently disabled rather than rejected at config load so an operator who turns reversible PII on partway through a rollout does not break their config. Re-enable semantic caching by removing reversible from every rule on that origin, or by moving the reversible workload to a separate origin without a semantic cache.

Two per-sink profiles ship, selected with the profile: field on a sink entry (see Sinks):

• internal applies the denylist above. Allows agent_id, tenant_id, JA3/JA4, request paths.
• external applies the denylist plus extra redactions: JA3/JA4 fingerprints and raw query strings.

These two names are the whole vocabulary today. Operator-defined named profiles (a profiles: block with custom deny lists and path globs) are a design idea, not a shipped key; a profiles: entry under observability.log is rejected at config compile as an unknown key. To extend redaction beyond the two built-ins, use the redact.fields: / redact.patterns: / redact.pii: blocks above, which are the shipped extension surface.

Enabling the redaction tests

The redaction contract is regressed by e2e/tests/redaction.rs. To run it locally:

cargo test -p sbproxy-e2e --release --test redaction

The test injects fixture inputs covering every member of the typed RedactedField enum, exercises every emitter (access, error, audit, trace), and asserts the marker appears in every sink while the original value appears in none of them. A failure is a CI block; redaction is the line we don't cross.

Traces

Tracer setup

OpenTelemetry SDK, pinned to the 0.27.x family. The tracer is initialized once at boot from proxy.observability.telemetry in sb.yml (see "Configuration" above).

OTLP gRPC (port 4317) is the default exporter. HTTP/protobuf (port 4318) is supported for environments that block gRPC. The stdout exporter is for local debugging only.

W3C TraceContext propagation

Every inbound HTTP path extracts traceparent and tracestate from request headers; if absent, a fresh root span starts. On the proxied path, the upstream request filter injects the distributed-tracing headers before the request leaves for the upstream, so proxied traffic propagates context end to end.

Extending that injection to every helper HTTP client the proxy owns (the ledger redeem client, the Web Bot Auth directory fetcher, outbound webhook delivery, OAuth token exchange, and the future payment-rail clients) is a design-stage contract: those clients do not inject traceparent today, and there is no CI gate enforcing it yet.

Span naming

Span names follow sbproxy.<pillar>.<verb>:

Span	Pillar
sbproxy.intake.accept	Top-level inbound request (root)
sbproxy.policy.enforce	Per-policy execution
sbproxy.action.challenge	Issue 402 challenge
sbproxy.action.redeem	Verify presented token / receipt
sbproxy.ledger.redeem	Outbound HTTP call to ledger
sbproxy.rail.settle	Outbound payment-rail settlement
sbproxy.transform.shape	Content transform
sbproxy.audit.emit	Append audit-log entry
sbproxy.notify.deliver	Outbound webhook delivery

Span attributes include the OTel semantic conventions (http.request.method, http.response.status_code, server.address) plus the SBproxy-specific set (sbproxy.request_id, sbproxy.tenant_id, sbproxy.route, sbproxy.agent_id, sbproxy.agent_class, sbproxy.rail, sbproxy.shape, sbproxy.ledger.idempotency_key).

High-cardinality attributes (request_id, agent_id) are span attributes only, never Prometheus labels.

AI gateway spans (gen_ai / OpenInference)

The AI request span (ai.request) follows the OpenTelemetry GenAI semantic conventions (gen_ai.*) and dual-emits the OpenInference (llm.*) vocabulary, so LLM-native trace backends render a full generation without remapping. Per request it carries:

Concept	gen_ai	OpenInference
Provider / model	gen_ai.system, gen_ai.request.model, gen_ai.response.model	llm.provider, llm.model_name
Request controls	gen_ai.request.temperature, gen_ai.request.max_tokens, gen_ai.request.top_p	n/a
Response identity	gen_ai.response.id, gen_ai.response.finish_reasons	n/a
Tokens (with cache + reasoning split)	gen_ai.usage.input_tokens, gen_ai.usage.output_tokens, gen_ai.usage.cache_read_tokens, gen_ai.usage.cache_write_tokens, gen_ai.usage.reasoning_tokens	llm.token_count.prompt, llm.token_count.completion, llm.token_count.total
Derived USD cost	sbproxy.ai.cost_usd_micros, gen_ai.usage.cost	llm.usage.total_cost
Pricing catalog revision	sbproxy.ai.pricing_version	n/a
Content (opt-in)	role-aware gen_ai.*.message span events	input.value, output.value, llm.input_messages.*, llm.output_messages.*
Failure	otel.status_code = ERROR plus error.type (guardrail_blocked, rate_limited, content_filter, budget_exceeded, upstream_5xx, timeout; generic dispatch failures use provider_error)	n/a
Tenant	sbproxy.tenant_id	n/a

Token counting happens at the proxy (not trusted from the upstream's self-report), cost is derived from the catalog stamped in sbproxy.ai.pricing_version, and the exact span value is sbproxy.ai.cost_usd_micros in micro-USD (1e-6 USD). The GenAI attribute set is pinned by a conformance test to OpenTelemetry GenAI semconv 1.36.0, with OpenInference pinned to a source revision in crates/sbproxy-ai/src/tracing_spans.rs, so emitted spans cannot silently drift off-spec.

To intentionally bump the AI span vocabulary, update the semconv constants and required field lists in crates/sbproxy-ai/src/tracing_spans.rs, update the span helpers for any renamed attributes, update this table, then run the span conformance test and the OTLP span-arrival e2e tests. Do not change these names just because the upstream experimental GenAI conventions moved; keep the existing emitted vocabulary until SBproxy explicitly ships an opt-in or migration.

Prompt and completion content capture is disabled unless the AI origin sets trace_content: true. When enabled, content is redacted with the secret redactor and the origin PII redactor when configured, capped at 8 KiB per captured value, and truncated with ...[truncated]; streaming completions are assembled from forwarded chunks before export.

Verified backend matrix

OTLP is vendor-agnostic. Use an OpenTelemetry Collector as the default ingress when you want fan-out, retries, memory limits, or backend-specific auth. Direct export is fine for a single trusted backend that accepts the same transport SBproxy is configured to emit and does not require custom OTLP headers. SBproxy's telemetry block exposes endpoint, transport, service name, resource attributes, sampling, and metric-export toggles; it does not expose per-exporter auth headers. Put Datadog Cloud, Honeycomb, Langfuse Cloud, and any other API-key backend behind the Collector.

The reference Compose stack under examples/observability-stack/ is the verified local path. SBproxy sends OTLP gRPC to the Collector on host port 4327; the Collector receives on container port 4317 and fans traces to Tempo, Phoenix, and Langfuse. It mirrors OTLP metrics to Prometheus with remote write and sends OTLP logs to Loki.

Backend	SBproxy endpoint	Collector exporter / backend endpoint	What renders
Arize Phoenix	http://otel-collector:4317 via the reference Collector, or direct http://localhost:6006 with transport: http when no Phoenix auth header is required	otlphttp/phoenix with endpoint: http://phoenix:6006 and x-project-name: SBproxy LLM Traces	LLM trace tree, provider, model, prompt, completion, token split, cost, latency, and status from gen_ai.*, OpenInference llm.*, input.value, and output.value.
Langfuse	http://otel-collector:4317; use the Collector for Cloud and authenticated self-hosted deployments	otlphttp/langfuse with endpoint: http://langfuse-web:3000/api/public/otel, Basic auth, and x-langfuse-ingestion-version: 4	LLM generation view with prompt, response, usage, cost, model, user/session metadata when supplied, and errors. Langfuse is HTTP OTLP only.
Jaeger	http://otel-collector:4317, or a Jaeger collector with OTLP enabled on 4317 gRPC / 4318 HTTP /v1/traces	otlp/jaeger to jaeger-collector:4317	Generic distributed traces. AI fields appear as searchable span attributes, but Jaeger does not render a specialized LLM trajectory UI.
Grafana Tempo	http://otel-collector:4317	otlp/tempo to tempo:4317; the reference stack wires this already	Generic traces in Grafana Explore and TraceQL. Use exemplars to jump from Prometheus outliers to traces.
Grafana Mimir	http://otel-collector:4317 when export_metrics: true, or Prometheus scrape plus remote write	prometheusremotewrite to http://<mimir-endpoint>/api/v1/push	Metrics panels for request rate, tokens, cost, cache, guardrail, and budget series. Mimir stores metrics, not traces; pair it with Tempo for the trace view.
Datadog	Datadog Agent on http://datadog-agent:4317 gRPC or http://datadog-agent:4318 HTTP; use a Collector or Datadog Distribution of the OTel Collector for cloud-auth export	Datadog Agent OTLP receiver, Datadog Distribution of the OTel Collector, or direct OTLP intake from a Collector	APM traces with gen_ai.*, llm.*, error.type, token, and cost attributes. Use Datadog dashboards or notebooks for LLM-specific panels.
Honeycomb	http://otel-collector:4317; use the Collector so it can attach the Honeycomb API-key header	otlphttp/honeycomb with x-honeycomb-team: ${HONEYCOMB_API_KEY}	High-cardinality trace exploration. request_id, agent_id, prompt capture, status, token, and cost attributes stay queryable without turning them into Prometheus labels.
Generic OTLP collector	http://otel-collector:4317 for gRPC or http://otel-collector:4318 for HTTP/protobuf	Any OTLP-compatible exporter chain	Whatever the downstream exporter supports. This is the best path for vendor migration and dual shipping.

SBproxy to Collector

Use this when the Collector is on the same Docker network as SBproxy:

proxy:
  observability:
    telemetry:
      enabled: true
      endpoint: "http://otel-collector:4317"
      transport: grpc
      service_name: "sbproxy"
      sample_rate: 0.1
      always_sample_errors: true
      keep_over_budget_usd: 1.00
      keep_slower_than_secs: 2.0
      export_metrics: true
      metrics_interval_secs: 30

Use this when SBproxy runs on the host and the reference Compose stack is running:

OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4327 \
  sbproxy serve --config sb.yml

There is no --metrics-listen flag; /metrics is served on the proxy listener itself (and on the admin listener when the admin API is enabled).

The reference Collector fan-out is:

exporters:
  otlp/tempo:
    endpoint: tempo:4317
    tls: { insecure: true }
  otlphttp/phoenix:
    endpoint: http://phoenix:6006
    headers:
      x-project-name: "SBproxy LLM Traces"
  otlphttp/langfuse:
    endpoint: http://langfuse-web:3000/api/public/otel
    headers:
      Authorization: "Basic ${env:LANGFUSE_OTEL_BASIC_AUTH}"
      x-langfuse-ingestion-version: "4"

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, tail_sampling, batch]
      exporters: [otlp/tempo, otlphttp/phoenix, otlphttp/langfuse]

Add a backend

Append one of these exporters to the traces or metrics pipeline in your Collector.

Jaeger:

exporters:
  otlp/jaeger:
    endpoint: jaeger-collector:4317
    tls: { insecure: true }

Grafana Mimir for OTLP metrics:

exporters:
  prometheusremotewrite:
    endpoint: http://mimir:9009/api/v1/push

Datadog Agent OTLP receiver:

otlp_config:
  receiver:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318
  logs:
    enabled: false

Honeycomb:

exporters:
  otlphttp/honeycomb:
    endpoint: https://api.honeycomb.io
    headers:
      x-honeycomb-team: ${env:HONEYCOMB_API_KEY}

For HTTP exporters, signal-specific paths are appended by the SDK or Collector when you configure the base OTLP endpoint. If you configure a traces-only endpoint directly, use the backend's /v1/traces path where required. Set transport: grpc for 4317 endpoints and transport: http for 4318 or HTTP/protobuf endpoints.

LLM trajectory check

Turn on content capture for the AI origin you are testing:

origins:
  "ai.local":
    action:
      type: ai_proxy
      providers:
        - name: openai
          api_key: ${OPENAI_API_KEY}
      trace_content: true

Then send one chat request. A healthy LLM-native backend shows a trace shaped like this:

trace: 9ff0a9a1c66e4c41ad3f2a8515d9d025
span: ai.request
attributes:
  gen_ai.operation.name = chat_completions
  gen_ai.system = openai
  gen_ai.request.model = gpt-4o-mini
  gen_ai.response.model = gpt-4o-mini-2024-07-18
  gen_ai.usage.input_tokens = 19
  gen_ai.usage.output_tokens = 23
  gen_ai.usage.cost = 0.000014
  llm.provider = openai
  llm.model_name = gpt-4o-mini
  llm.token_count.prompt = 19
  llm.token_count.completion = 23
  llm.token_count.total = 42
  llm.usage.total_cost = 0.000014
  sbproxy.ai.cost_usd_micros = 14
  sbproxy.ai.pricing_version = 2026-06-01
  sbproxy.tenant_id = default
  input.value = "Say hello from SBproxy observability."
  output.value = "Hello from SBproxy observability."
events:
  gen_ai.user.message
  gen_ai.assistant.message

On a blocked or failed generation, otel.status_code = ERROR and error.type is one of guardrail_blocked, rate_limited, content_filter, budget_exceeded, upstream_5xx, or timeout; generic dispatch failures use provider_error. Phoenix, Langfuse, Datadog, Honeycomb, Jaeger, and Tempo all preserve those attributes. The difference is presentation: Phoenix and Langfuse render a generation view, while the generic trace backends expose the same fields as searchable attributes.

Sampling

Wave 1 ships head-based sampling, evaluated at the root span:

1. If the inbound traceparent has the sampled bit set, sample (parent-based).
2. Else if the request errors (5xx, policy block, ledger denial), sample 100%.
3. Else sample at sample_rate (default 0.1; see the telemetry: block above).

Tail-based sampling (drop based on outcome at span end) is deferred to Wave 6. The reference Compose stack ships an OTel Collector recipe operators can opt into; the proxy itself does not run a tail sampler.

Exemplars

Exemplars are wired on every histogram where "click the outlier in Grafana, get the trace" is a high-value debugging path:

• sbproxy_request_duration_seconds_bucket (top-level latency)
• sbproxy_ledger_redeem_duration_seconds_bucket (ledger tail)
• sbproxy_policy_evaluation_duration_seconds_bucket (policy regressions)
• sbproxy_outbound_request_duration_seconds_bucket (per-outbound tail)
• sbproxy_audit_emit_duration_seconds_bucket (audit-log write tail)

Exemplars carry trace_id per scrape interval. Prometheus needs --enable-feature=exemplar-storage; the reference stack sets it.

Dashboards

JSON files live under deploy/dashboards/:

• overview.json - request rate, error rate, latency p95/p99, ledger health.
• per-agent.json - per-agent_class and per-agent_vendor request rate, redeem rate, revenue (Wave 2 fills the revenue panel).
• policy-triggers.json - per-policy block rate, decision distribution.
• audit-log.json - admin-action volume, outcome distribution, append-only verification status.
• traces-overview.json - span chain length, slowest spans, sampling rate.

The Helm chart ships them as a single ConfigMap that the kiwigrid Grafana sidecar discovers by label. The chart's values are:

# values.yaml
dashboards:
  enabled: true
  # Label key the Grafana sidecar watches; override if your sidecar
  # uses a different label key.
  sidecarLabel: grafana_dashboard

Set dashboards.enabled: false to skip the ConfigMap when dashboards are managed out of band. Operators who run Grafana outside Helm can kubectl create configmap the JSON files from deploy/dashboards/ directly with the grafana_dashboard=1 label.

Alerts

Three tiers, each with explicit on-call semantics:

• Page (P1, immediate human action). Goes to PagerDuty; on-call acks within 15 minutes. Examples: ledger down, audit-log write failure, rail quorum loss, restore-drill miss.
• Ticket (P2, next business day). Files an issue in the on-call queue. Examples: latency p95 sustained breach, webhook delivery failure rate, classifier drift (Wave 5).
• Log-only (P3). Records the alert in Alertmanager but routes to log destinations only. Examples: cardinality near budget (90% of cap), deprecated-flag use, exemplar emission rate dropping.

Burn-rate windows for the page tier: 5m AND 1h at 14.4x, 30m AND 6h at 6x. Ticket tier: 2h AND 24h at 3x. Each paging alert carries a runbook_id label so on-call has a stable correlation key into deployment-specific runbooks.

Health endpoints

The proxy listener itself serves /health (and /metrics), so a fresh install with no extra configuration is probeable. /healthz and /readyz live on the admin listener, which is disabled by default and binds 127.0.0.1:9090 when enabled:

curl http://localhost:8080/health
# 200 OK, {"status":"ok"}. Served on the proxy listener.

curl http://localhost:9090/healthz
# 200 OK, no body. Liveness only; the kubelet uses this to decide whether to restart the pod.

curl http://localhost:9090/readyz
# 200 OK with a JSON body listing each component status.
# 503 with the same body when any required dependency is unhealthy.

There is no separate management port 9091.

/readyz reports per-component status. The registered components are ledger (redeem recency), bot_auth_directory (directory freshness), agent_registry, mesh_quorum, and telemetry_sink (a poisoned sink dispatcher fails readiness so the load balancer drains a telemetry-blind instance). A component whose feature is not configured reports not_configured and passes readiness.

Reference Compose stack

examples/observability-stack/ boots Prometheus, Grafana, Tempo, Loki, Phoenix, Langfuse, and an OTel Collector with one command:

cd examples/observability-stack
docker compose up -d

Then open:

• Grafana at http://localhost:3000 (login admin / admin)
• Prometheus at http://localhost:9090
• Loki ready endpoint at http://localhost:3100/ready
• Tempo via Grafana (no first-class UI)
• Phoenix at http://localhost:6006, project SBproxy LLM Traces
• Langfuse at http://localhost:3001 (login admin@sbproxy.local / sbproxy-local-admin), project SBproxy LLM Traces

Point SBproxy at the stack:

OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4327 \
  sbproxy serve --config sb.yml

The proxy exposes /metrics on its own listener (proxy.http_bind_port), so with the default 8080 the Compose Prometheus job scrapes host.docker.internal:8080. Adjust the scrape target if your proxy binds a different port.

The OTLP endpoint targets the OTel Collector (host port 4327, mapped to the container's 4317). The collector fans traces to Tempo, Phoenix, and Langfuse, mirrors OTLP metrics to Prometheus, and sends OTLP logs to Loki. The dashboards from deploy/dashboards/ are pre-provisioned in Grafana, so you see metrics, logs, and traces flow as soon as the proxy starts handling requests.

For a full LLM-native smoke test, enable trace_content: true on the AI origin and send a chat-completions request through SBproxy. Phoenix and Langfuse render the same generation with prompt, response, provider, model, token split, USD cost, TTFT, latency, and status fields from the emitted gen_ai.* and OpenInference attributes/events.

docker compose down -v drops the named volumes for Prometheus, Grafana, Tempo, Loki, and Langfuse's Postgres, ClickHouse, MinIO, and Redis storage for a fresh start.

See also

• audit-log.md - admin-action audit envelope.
• ai-crawl-control.md - per-agent observability for the Pay Per Crawl policy.
• deploy/dashboards/ - Grafana JSON for the Wave 1 panels.
• deploy/alerts/ - PromQL recording and alerting rules.
• examples/observability-stack/ - the reference Compose stack.