Memory & GC

Metrics source: dsh.runtime.memory{area} is exposed by this plugin; dsh_v8js_memory_* / dsh_v8js_gc_* come from the community package @opentelemetry/instrumentation-runtime-node (dependency). Related: Event Loop Delay (GC is the top suspect for p99 long pauses) · README

---

1. Two-Layer Perspective

Layer	Metric	Prometheus Name	Question Answered
Process	dsh.runtime.memory{area}	dsh_dsh_runtime_memory_bytes	OS perspective: how much this process uses (RSS/heap/external)
V8	dsh_v8js_memory_heap_space_size_bytes{v8js_heap_space_name}	Same name	Engine perspective: per-space heap usage, key for leak localization
Pressure	dsh_v8js_memory_heap_space_available_size_bytes	Same name	Available allocation per space, OOM precursor signal
Cost	dsh_v8js_gc_duration_seconds_*	Histogram	GC frequency and pause, converts memory issues to latency cost

area / space Field Reference

Plugin {area}: rss (resident memory, includes external), heap_used / heap_total (V8 heap), external (Buffer and other non-heap), array_buffers.

V8 spaces to focus on: old_space (old generation, main battleground for leaks — surviving objects accumulate here), new_space (young generation, frequent Scavenge collection), large_object_space (large objects >256KB Buffer/arrays go directly here).

2. Diagnosis Matrix

Signal	Threshold Reference	Meaning
RSS stable	Smooth fluctuation	Healthy
RSS step-increase without drop	Monotonic increase over days	Leak or unbounded cache: cross-check heap_used — synchronized growth = heap leak; heap flat but RSS growing = external/ArrayBuffer leak (Buffer not released)
heap_used / heap_total	Sustained > 90%	Heap pressure, increased GC stress
old_space available	Approaching 0	OOM precursor (this community metric is the preferred heap pressure alert)
GC major frequency	Noticeably increasing / longer duration	Frequent old-gen collection = leak or heap config too small
GC p95 pause	> 50ms	Starting to consume latency budget; > 200ms directly creates event loop p99 spikes

3. Diagnosis Example

Instance	RSS	heap_used/total	old_space Usage	GC (1h major count / p95 pause)	Assessment
dsh-agent	1.73GB	608MB / 722MB	544MB	~720 / 19.3ms	Heap stable, GC frequent but pauses small, healthy
dsh-agent-web2	2.22GB	1.51GB / 1.55GB (98%)	461MB, but large_object_space at 1.03GB	19 / 9.4ms	🔴 Two risk points (see below)

web2 two concerns:

1. heap_used/total = 98%: Heap nearly full. GC pauses currently small (p95 9.4ms), but old-gen at 98% utilization means each major GC runs at the limit; further heap_total growth triggers "allocate-then-GC" death spiral
2. large_object_space 1.03GB: Nearly 2/3 of heap is large objects (large Buffers/JSON strings/arrays going directly to large-object space). The correlation with event loop p99 ≈3s is a key clue — large object creation/serialization/copying occupies both heap and main thread, high probability that a single root cause explains both symptoms (Pyroscope heap profile can confirm)

4. Remediation Playbook

1. Classify: RSS ↑ + heap_used ↑ = heap issue; RSS ↑ + heap flat = check external/array_buffers (Buffer not released / native modules)
2. Locate: Grafana → Pyroscope → service_name=<instance> → Heap profile (production-ready, non-invasive) sorted by retained size to find holders
3. Short-term mitigation: Restart target process to release heap (if GUI host, restarting kills active sessions — requires maintenance window)
4. Root fix: Large object chunked streaming, explicit caches with LRU limits, check global arrays for grow-only pattern
5. Verify: old_space usage drops + GC major frequency drops + event loop p99 drops (all three must move in same direction to confirm fix)

5. Common Queries

Prometheus metric names may vary based on Collector transformation — verify with actual deployment.

# Process memory overview (MB)
dsh_dsh_runtime_memory_bytes / 1024 / 1024

# Heap utilization ratio (approaching 1 = nearly full)
dsh_v8js_memory_heap_used_bytes / dsh_v8js_memory_heap_total_bytes

# old_space leak trend (6h linear extrapolation; slope > 0 sustained = leak signal)
deriv(dsh_v8js_memory_heap_space_size_bytes{v8js_heap_space_name="old_space"}[6h])

# GC pause p95 and frequency
histogram_quantile(0.95, sum by (le, exported_job) (rate(dsh_v8js_gc_duration_seconds_bucket[1h])))
sum by (exported_job, v8js_gc_type) (increase(dsh_v8js_gc_duration_seconds_count[1h]))

# External suspicious growth (check when RSS grows but heap doesn't)
dsh_dsh_runtime_memory_bytes{area=~"external|array_buffers"} / 1024 / 1024