Memory Limiter - Phase 1

This document describes the Phase 1 implementation of the process-wide memory limiter. It covers current behavior only. The longer-term hierarchical lease-and-ticket design is planned for a separate document.

For a status-labelled map of process memory views, allocator telemetry, retained-work attribution, throttling, and control actions, see Memory Resource Management.

Problem

The collector already has bounded channels, topic publish limits, and receiver-side backpressure. Those controls are local: each one protects a single queue or subsystem, but nothing enforces a shared RAM ceiling across:

• concurrent receiver ingress
• decoded and buffered request bodies
• multiple queues and topics simultaneously
• allocator overhead and fragmentation
• retained protocol state

Phase 1 adds a process-wide guardrail against sustained memory pressure.

Scope

Phase 1 is a process-wide observed-memory limiter implemented as an engine service. It samples actual process memory on a fixed interval and gates receiver ingress based on the result.

Implementation note:

• Sampling and pressure classification remain process-wide in the controller.
• On pressure transitions, the controller propagates updates to receivers through the pipeline control plane.
• These transitions are delivered as receiver control messages (NodeControlMsg::MemoryPressureChanged).
• Each receiver maintains receiver-local admission state and consults that local state on ingress hot paths.

What it does:

• Sample process memory on a configurable interval
• Classify pressure as Normal, Soft, or Hard
• Keep Soft informational for the memory limiter itself; optional receiver admission policies may use it as an input signal
• Shed ingress at the receiver boundary only under Hard (in enforce mode)
• Optionally fail the readiness probe under Hard (in enforce mode)
• Optionally run in observe_only mode for metrics and logs without enforcement
• Expose process-level memory and pressure metrics

What it does not do:

• Per-pipeline memory budgets
• Ticketed byte accounting
• Per-core local leases
• Queue or topic byte charging
• Reclaim hooks for stateful components
• OTAP stream recycling

Why This Complements Bounded Channels

Bounded channels and topic policies are not replaced by the memory limiter. They serve different purposes:

• Bounded channels / topics control local backlog growth within one queue
• Memory limiter controls total process memory at the outer ingress boundary

When an internal queue fills, cooperative producers block or drop according to that queue's policy. When the process as a whole approaches its memory limit, a separate ingress policy is needed at the boundary - one that does not depend on knowing which internal queue is the cause.

A key difference in timing: bounded channels react after a message has already been accepted, decoded, and buffered. The memory limiter acts earlier - at the receiver ingress boundary, before expensive body accumulation or downstream admission. This means the limiter can shed load without the full cost of accepting the work first.

Configuration

The memory limiter is configured under policies.resources.memory_limiter in the engine config. This field is supported only at the top-level policies scope; group and pipeline overrides are rejected during validation.

policies:
  resources:
    memory_limiter:
      mode: enforce        # or observe_only for metrics/logs without rejection
      source: auto          # prefer auto/cgroup on Linux containers
      check_interval: 1s    # minimum 100ms
      soft_limit: 7 GiB     # if set explicitly, keep headroom above idle
      hard_limit: 8 GiB     # shedding threshold; not a strict cap
      hysteresis: 512 MiB   # bytes below soft_limit required to leave Soft
      retry_after_secs: 5   # used only in enforce mode
      fail_readiness_on_hard: true  # used only in enforce mode
      purge_on_hard: false  # optional jemalloc purge hook, disabled by default
      purge_min_interval: 5s

Limit selection

Limits are resolved in this order:

1. Explicit soft_limit and hard_limit (both required if either is set)
2. source: auto with cgroup-derived limits when a cgroup memory controller is detected

When source: auto is used and no explicit limits are configured, the limiter reads the cgroup hard cap (memory.max on cgroup v2, memory.limit_in_bytes on v1) and derives both thresholds from it:

• soft_limit = 90% of the cgroup limit
• hard_limit = 95% of the cgroup limit

This leaves a 5% buffer between the limiter's shedding threshold and the kernel's actual OOM kill boundary.

When no cgroup memory controller is detected (for example on macOS, Windows, or a bare-metal Linux process without a memory cgroup), auto does not fall back to deriving limits from total physical RAM. Explicit soft_limit and hard_limit must be provided, or startup fails with a configuration error. This is intentional: silently deriving limits from total host RAM could produce dangerously high thresholds on large machines.

Sizing guidance

• Do not set soft_limit and hard_limit only a few MiB above observed idle RSS. Small run-to-run variance can be enough to flip the limiter from "never reaches Hard" to "enters Hard and stays there".
• Treat hard_limit as an ingress-shedding threshold, not as a promise that process memory will stay below that value. The limiter is periodic and reactive, so bursty workloads can overshoot it before shedding takes effect.
• For RSS-based configurations, size limits with explicit headroom above sustained steady-state memory, not just above an idle snapshot. As a rule of thumb, start with at least 15-20% headroom above observed steady-state memory, then adjust using production measurements.
• For Linux containerized deployments, prefer source: auto so the limiter can derive cgroup-based limits instead of relying on RSS alone.
• Auto-derived cgroup limits use 90%/95% of the cgroup cap. If you want the limiter to begin shedding well before the container approaches its memory limit, configure explicit soft_limit and hard_limit values relative to expected peak working-set usage.
• Recovery from Hard requires usage to fall below soft_limit, not merely below hard_limit. In practice, soft_limit - steady_state_usage is the real recovery headroom. If soft_limit is set below the process's irreducible working set, Hard becomes a permanent state.

Mode

• mode is required when memory_limiter is configured. This is an explicit operator choice, not an implicit default.
• mode: enforce sheds ingress under Hard pressure and can fail readiness.
• mode: observe_only keeps classification, logs, and metrics enabled but suppresses ingress shedding, readiness failure, and forced jemalloc purge.
• For a first rollout, prefer mode: observe_only so you can validate sampled usage, pressure transitions, dashboards, and readiness behavior before enabling enforcement.

Mode is set at startup and cannot be changed while the process is running.

Purge

• purge_on_hard is an optional jemalloc-only mitigation for RSS-based retention. When enabled, a tick whose pre-purge sample classifies as Hard attempts a forced jemalloc purge, then re-samples memory before classifying the next state.
• purge_min_interval controls the minimum time between purge attempts. Both successful and failed attempts count toward the rate limit, which prevents the limiter from spamming a broken purge call on every tick.
• Purge is best-effort. If the purge call or the post-purge re-sample fails, the limiter logs a warning (process_memory_limiter.purge_failed) and commits the pre-purge Hard classification. A purge failure never prevents the limiter from updating shared state.
• purge_on_hard is ignored in observe_only mode. If purge_on_hard is enabled but jemalloc purge support is not available in the build, the limiter logs a startup warning (process_memory_limiter.purge_unavailable) and continues without purge.
• Keep purge_on_hard disabled unless you have validated it on your workload. It is intended as an escape hatch for allocator-retained resident pages, not as the default recovery mechanism.

Memory source

Source	Description
auto	Cgroup working set if available, otherwise RSS, otherwise jemalloc resident
cgroup	Cgroup working set (v1 and v2 supported); fails if no cgroup controller detected
rss	Process RSS via the memory_stats crate
jemalloc_resident	jemalloc resident bytes; requires jemalloc feature

Cgroup sampling subtracts inactive_file pages from the raw usage counter, consistent with how container orchestrators report memory usage.

Source selection guidance

• Prefer source: auto for Linux containerized deployments. In Kubernetes and other cgroup-managed environments, this usually resolves to cgroup.
• cgroup is not Kubernetes-specific. It is also meaningful for Linux services that run inside a memory-constrained systemd slice or another configured cgroup.
• For a plain Linux process started from a shell without a meaningful cgroup memory limit, explicit soft_limit and hard_limit are typically required.
• jemalloc_resident is a resident-memory signal, not a live-allocation signal. It is not a recovery-oriented alternative to rss.

Platform Support

The Phase 1 limiter architecture is portable, but the current implementation is strongest on Linux.

• Linux has the best support today because auto can use cgroup working-set sampling and cgroup-derived limits.
• On non-Linux platforms, the limiter can still use RSS-based sampling, and it can use explicit configured limits.
• auto does not fall back to total physical RAM on macOS or Windows. If no cgroup limit is available and no explicit limits are set, startup fails.

Internally, the implementation keeps the platform-specific logic at the memory probe boundary. The pressure-state logic, receiver shedding behavior, and controller integration remain platform-independent.

Recovery caveat for rss

When the collector is built with jemalloc and configured with source: rss, recovery after burst load can be slow. Freed allocations may remain resident in jemalloc arenas for reuse, so process RSS can stay above hard_limit even after live workload drops. In that state the limiter may continue to reject new ingress until resident pages are released or the process restarts.

For containerized Linux deployments, prefer source: auto or source: cgroup when available. Resident-memory sources are conservative protection signals, but they are not ideal recovery signals after allocator-heavy bursts.

Container deployment notes

• In containers, source: auto should resolve to cgroup and generally provides better recovery behavior than rss.
• If you rely on /readyz from Kubernetes or another orchestrator, bind the admin server to a pod-reachable address such as --http-admin-bind 0.0.0.0:8080. The default loopback bind is not sufficient for external readiness probes.
• Set --num-cores in line with the container CPU limit. Otherwise the engine may start more worker threads than the container is intended to run, which increases idle memory overhead.

Pressure Semantics

The limiter maintains a three-level pressure state:

Level	Meaning	Receiver behavior
Normal	Below soft_limit	No action
Soft	Above soft_limit	Observed; receiver policies may react
Hard	Above hard_limit	Ingress shedding enabled (enforce mode only)

When mode: observe_only is configured, the same state transitions still occur, but Hard remains advisory: receivers continue accepting requests and the readiness endpoint stays healthy.

stateDiagram-v2
    [*] --> Normal

    Normal --> Soft   : usage >= soft_limit
    Normal --> Hard   : usage >= hard_limit

    Soft --> Hard     : usage >= hard_limit
    Soft --> Normal   : usage < soft_limit - hysteresis

    Hard --> Soft     : usage < soft_limit

Transitions

• Escalation (Normal -> Soft, Soft -> Hard, Normal -> Hard) is immediate when the threshold is crossed.
• Recovery from Soft requires usage to drop below soft_limit - hysteresis before returning to Normal. This prevents oscillation when usage hovers near the soft threshold. When hysteresis is omitted it defaults to min(hard_limit - soft_limit, soft_limit / 10), a narrow band so recovery happens once usage falls modestly below soft_limit rather than only after it collapses toward zero.
• Recovery from Hard requires usage to drop below soft_limit before returning to Soft.
• Phase 1 does not implement cooldown timers. Those are planned for a later phase.

Because sampling is periodic, the limiter can move directly from Normal to Hard under fast bursts without spending a full interval in Soft.

Operationally, this means soft_limit is the reopening threshold after shedding begins. hard_limit starts rejection, but recovery does not begin until usage has fallen below soft_limit.

Receiver Behavior Under Hard Pressure

Phase 1 applies protocol-native overload signals at each receiver. The following behaviors apply in enforce mode only. In observe_only mode, receivers continue accepting requests regardless of pressure level.

Receiver	Hard-pressure behavior
OTLP HTTP	503 Service Unavailable with Retry-After: <retry_after_secs> header
OTLP gRPC	RESOURCE_EXHAUSTED with grpc-retry-pushback-ms: <retry_ms> metadata
OTAP gRPC stream open / next-read boundary	RESOURCE_EXHAUSTED + grpc-retry-pushback-ms before stream admission, and for already-open streams at the next read boundary
OTAP gRPC per-batch	ResourceExhausted in the OTAP Arrow batch status (ArrowStatus code 8)
Syslog / CEF TCP	Accept then immediately drop new connections; close active connections mid-stream
Syslog / CEF UDP	Drop incoming datagrams

Soft pressure: the memory limiter does not reject requests solely because the process is above the soft limit. The engine-level memory_pressure_state metric reflects 1 (Soft) and process_memory_usage_bytes reflects the elevated usage. A process_memory_limiter.transition log event is emitted at info level on entry to Soft. Optional receiver admission policies, such as pressure-aware rate throttling, may use Soft as their activation signal. The behaviors in the table above apply only at Hard in enforce mode.

For the v1 pressure-aware rate policy, OTLP supports only request_bytes, measured as decompressed OTLP payload bytes at the receiver admission point. V1 does not count OTLP telemetry items. Syslog / CEF supports messages, measured as one UDP datagram or one emitted TCP record before parsing. A normal TCP line is one record; a line that exceeds MAX_MESSAGE_SIZE may be emitted as multiple bounded-read fragments, and each emitted fragment is counted separately. The current policy does not measure wire bytes. Named limiter declarations are preserved through policy resolution. A receiver can bind one limiter with rate_limiters: [name] or opt out with rate_limiters: []. Omitting the field also leaves the receiver unbound. Mixed receiver pipelines can bind OTLP and Syslog / CEF receivers to limiters with different units.

The rate state is receiver-local and lock-free. It uses a token-bucket-equivalent GCRA state machine with bounded debt, so each receiver instance can continue tracking over-limit traffic while memory is normal and apply that state when soft pressure begins. OTLP keeps decompressed-byte accounting as the authoritative charge. The first request that crosses the limit is therefore collected and decompressed before its exact charge is known, within the receiver's existing request-size bound. A non-charging exhausted-bucket check rejects subsequent requests before body collection or gRPC message assembly while active enforcement and exhaustion continue.

The configured interval also determines the burst window. For example, allow: 100, interval: 60s, and burst: 100 produce a 60-second burst window and a debt ceiling of two burst windows. Normal-pressure observation can reach that ceiling. If pressure then activates, a charge of weight w may take up to one burst window plus the rate cost of w to conform. The worst case for any admissible charge is two burst windows. Use long intervals only when that lockout behavior is acceptable.

Once request weight is known, retryable rate-limit responses advertise the earliest whole-second bucket recovery delay for the refused weight. They do not reuse the memory limiter's sampling retry hint. Instance-wide fast refusal carries no retry guidance because request weight is unavailable at that boundary and the receiver cannot yet distinguish a transient refusal from a permanently oversized request.

An OTLP request larger than the configured burst can never fit the bucket while pressure gating is active. HTTP rejects it with 413 and no Retry-After; gRPC returns RESOURCE_EXHAUSTED with negative retry pushback. Configure burst at least as large as the largest request the receiver should accept during pressure.

V1 supports rate limiting only for OTLP and Syslog / CEF receivers. A component that names a limiter must bind it during construction or startup fails. Non-participating components remain unbound unless they explicitly select a limiter. rate_limiters: [] is an explicit opt-out.

The engine resolves the policy into a construction-time admission binder on the node's PipelineContext. A participating component binds exactly once, naming the dimension it measures and supplying its local or shared pressure state. The returned gate owns the receiver-instance bucket. Cloning that gate for multiple protocol stacks shares the bucket; cloning PipelineContext never creates or shares bucket capacity accidentally.

V1 state is local to each receiver instance. It provides receiver-instance rate isolation and pressure-triggered load shedding, not a group-wide budget, tenant scheduling, or fairness. Tenant-keyed limits and shared group/process state need additional keying, cardinality, routing, and scheduling designs.

Syslog / CEF client behavior under Hard pressure:

• TCP: The receiver accepts new connections and then immediately drops the socket, closing active connections mid-stream. The connection is closed at the transport layer with no application-level retry hint - unlike OTLP/OTAP, no Retry-After or pushback value is sent. Most syslog clients (rsyslog, syslog-ng, Fluent Bit) have their own reconnect backoff, but they have no signal about why the connection was closed or how long to wait before reconnecting.
• UDP: Datagrams are silently dropped at the receiver. UDP is fire-and-forget, so the sender receives no feedback at all. Events are permanently lost with no indication to the sending client. Operators relying on UDP syslog should treat Hard pressure as a potential data-loss event and monitor received_logs_rejected_memory_pressure to detect it.

Syslog / CEF behavior under pressure-aware rate throttling:

• TCP: Over-limit framed messages are dropped while the connection remains open. This is a silent message drop: plain syslog TCP has no per-message acknowledgement or retry hint, so the client does not know which line was dropped. If an oversized TCP fragment is over limit, remaining fragments from that oversized line are discarded through the newline. Hard memory pressure still closes active connections.
• UDP: Over-limit datagrams are silently dropped. UDP senders receive no feedback, so operators should monitor admission.rate_limiter.refusals{dimension="messages"} and distinguish enforced, observe-only, and permanently oversized outcomes with the refusal attribute.

Design rationale: explicit rejection is preferred over transport-level stalling. For TCP, holding large numbers of stalled open connections under pressure can consume more resources than the data they carry. Explicit close or refusal is observable, bounded, and gives senders a clear signal to back off.

Known gap: OTAP stream reads are checked for memory pressure at the next read boundary. If pressure flips to Hard while a stream task is already blocked in message().await, one additional batch may still be read before the stream is rejected.

Readiness Integration

When fail_readiness_on_hard is enabled (default: true), the /readyz endpoint returns 503 Service Unavailable while the limiter is in Hard pressure in enforce mode. In observe_only, readiness remains healthy even if pressure reaches Hard. The /livez endpoint is unaffected.

Metrics

Engine-level (emitted by the engine metrics monitor)

All engine metrics are registered under the engine metric-set.

Metric	Description
memory_rss	Current process RSS in bytes
process_memory_usage_bytes	Most recent memory limiter sample in bytes
process_memory_soft_limit_bytes	Effective soft limit in bytes
process_memory_hard_limit_bytes	Effective hard limit in bytes
memory_pressure_state	Current pressure level (0=Normal, 1=Soft, 2=Hard)
cpu_utilization	Process CPU utilization as a ratio in [0, 1], normalized across all system cores

Admission and receiver-level

Metric	Receiver	Description
receiver.otlp.rejections.requests{error.type="memory_pressure"}	OTLP (gRPC + HTTP)	Requests rejected due to memory pressure, partitioned by protocol
receiver.otlp.rejections.requests{error.type="rate_limit"}	OTLP (gRPC + HTTP)	Requests refused by rate throttling, partitioned by protocol
admission.rate_limiter.refusals	Any participating component	Admission attempts refused, oversized, or admitted in observe-only mode, partitioned by bounded dimension and refusal attributes
receiver.otap.rejections.streams{error.type="memory_pressure"}	OTAP gRPC	Streaming RPCs rejected due to memory pressure
receiver.otap.rejections.batches{error.type="memory_pressure"}	OTAP gRPC	Batches rejected due to memory pressure after stream admission
receiver.syslog_cef.connections.rejected	Syslog / CEF TCP	Connections rejected or closed
receiver.syslog_cef.rejections.items{error.type="memory_pressure"}	Syslog / CEF	Log records dropped under pressure

Structured log events

Event	Level	Description
process_memory_limiter.transition	info/warn	Emitted on every pressure level change. Hard transitions log at warn level.
admission.binding.summary	info	Emitted once per pipeline with bounded lists of nodes that bound or explicitly opted out of admission.
process_memory_limiter.purge	info	Emitted after a successful forced jemalloc purge. Includes pre/post usage and duration.
process_memory_limiter.purge_failed	warn	Emitted when a purge attempt or post-purge re-sample fails.
process_memory_limiter.purge_unavailable	warn	Emitted at startup when purge_on_hard is enabled but no allocator purge backend is available in this build.
process_memory_limiter.sample_failed	warn	Emitted when a periodic memory sample fails.
process_memory_limiter.observe_only_ignored_setting	warn	Emitted at startup when purge_on_hard: true is set with mode: observe_only (purge is suppressed in that mode).
syslog_cef_receiver.rate_limit.drop	warn	Emitted once per TCP connection when pressure-aware rate throttling first drops an over-limit message on that connection.

Tradeoffs

Phase 1 is deliberately simpler than the long-term design.

Benefits:

• Low implementation risk
• Additional process-wide protection against memory pressure
• Enforcement hot paths are receiver-local and NUMA-friendly; the process-wide sampler is not consulted on ingress
• Clean fit with existing receiver admission controls
• No invasive queue or pdata instrumentation required

Limitations:

• Reactive: detects pressure after memory is already allocated
• The configured hard_limit is a shedding threshold, not a strict cap on peak process memory
• Process-wide only: cannot isolate one misbehaving pipeline
• No accounting for bytes retained in queues, topics, or processor state
• No reclaim actions by default; optional purge_on_hard can force a jemalloc purge before reclassification

Relationship to Later Phases

Later phases will add:

• Queue and topic byte accounting
• Per-pipeline memory budgets
• Per-core local leases with bounded overshoot
• MemoryTicket ownership on retained work items
• Reclaim hooks for stateful components (batch processor, retry, durable buffer)
• OTAP stream-state accounting and recycling

Those are out of scope for this document and this branch.