Event Loop Delay

Metrics source: this plugin exposes dsh.runtime.event_loop.delay{quantile}; companion metrics dsh_nodejs_eventloop_delay_* come from the community package @opentelemetry/instrumentation-runtime-node (dependency). Related: ELU Guide (ELU saturation, the delay perspective complement) · README (plugin configuration)

---

1. Definition

Event Loop Delay = how long a timer callback was actually postponed from its scheduled execution time.

Node.js implementation: After setTimeout(fn, 1), measure the difference between when fn actually executes and when it was scheduled. When the event loop is idle, the difference ≈ 1ms + jitter; when the thread is occupied by previous tasks, the difference equals the occupation duration — so it directly measures how long the main thread was blocked in a single instance.

2. Two Implementations, Two Perspectives (Must Read Together)

Perspective	Plugin Metric	Prometheus Name	Characteristics
Single block duration	dsh.runtime.event_loop.delay{quantile}	dsh_dsh_runtime_event_loop_delay_seconds	quantile=0.5/0.9/0.99 three series; p99 directly answers "how long the worst round was blocked"
Rolling window percentile	dsh_nodejs_eventloop_delay_p50/p90/p99_*	Same name	Community package 1min rolling statistics; also mean/min/max/stddev

The two p99 values are expected to be close (same algorithm, different implementations) for cross-validation; the plugin version is better for "seeing real-time spike values", the community package version for "seeing the 1min window shape".

Division of labor with ELU: ELU says "busy or not" (proportion), delay says "how long blocked" (absolute duration). The web2 pattern = ELU 0.90 (busy) + p50 20ms (normally not blocked) + p99 2.9s (occasional large block).

3. Percentile Value Interpretation

Metric	Healthy	Watch	Action Line	Notes
p50	< 10ms	< 50ms	> 100ms	Steady state; sustained >100ms means chronic overload, not occasional
p90	< 50ms	< 100ms	> 200ms	Upper bound for normal load
p99	< 100ms	< 500ms	> 1s	Core alert position: single block >1s causes perceptible UI/streaming lag; evaluate in conjunction with plugin ELU degradation lines (0.90/0.95)

Thresholds are built into the Runtime Diagnostics dashboard (p99 + ELU overlaid on same chart).

4. Combined Diagnosis with GC / Memory

Priority order for identifying the three main suspects of p99 long blocks:

1. GC major (cross-reference memory-gc.md dsh_v8js_gc_duration_seconds_bucket p95): GC p95 and EL delay p99 same order of magnitude = block mostly caused by GC
2. Large synchronous computation/serialization (locate with Pyroscope Wall profile)
3. Synchronous I/O (check for *Sync calls in code)

5. Diagnosis Example

Instance	p50 / p90 / p99	Community Package mean / 1h peak	Assessment
dsh-agent	20ms / 21ms / 42ms	10.6ms / 33ms	✅ Healthy
dsh-agent-web2	20ms / 25ms / 2.95s	94ms / 3.34s	🔴 Periodic long blocks (sustained worsening: p99 from 2.87s → 2.95s)

web2: p50/p90 normal + p99 ≈3s = most requests experience good service, but a few hitting the block window experience extremely poor service; block source identification follows §4 priority (GC p95 only 9.4ms, rules out GC → points to large synchronous computation, locate with Pyroscope).

6. Remediation Playbook

1. p99 sustained >1s: Pyroscope Wall profile for service_name=<instance> to find the widest stack
2. Common culprits: large JSON parse/stringify, un-chunked large loops, synchronous crypto/FS, child_process.execSync
3. Chunked processing (yield thread with setImmediate every N ms) or move to worker threads
4. Post-fix verification: p99 drops in same window + ELU drops + telemetry export failures stop increasing (once blocks are cleared, export is no longer suppressed by ELU gating)

7. Common Queries

Prometheus metric names may vary based on Collector transformation — verify with actual deployment.

# All three percentiles together (overlay ELU for best insight)
dsh_dsh_runtime_event_loop_delay_seconds
dsh_dsh_runtime_event_loop_utilization_ratio

# 1h p99 peak (has second-level blocking occurred?)
max_over_time(dsh_nodejs_eventloop_delay_p99_seconds[1h])

# Duration p99 exceeded 1s (at 10s sampling, max is 360)
count_over_time((dsh_dsh_runtime_event_loop_delay_seconds{quantile="0.99"} > 1)[1h:])

# Bad neighbor detection: p99 ranking across all instances on same host
topk(5, max_over_time(dsh_nodejs_eventloop_delay_p99_seconds[10m]))