GMS Rate Limiting

August 7, 2026 · View on GitHub

This guide explains how to enable, configure, observe, and troubleshoot GMS HTTP service rate limiting — limits on incoming API traffic to GMS (GraphQL, OpenAPI, Rest.li, native auth routes). It protects the Metadata Service from overload and caps abuse on sensitive endpoints such as /auth/signUp.

It also documents MCP / Kafka ingest throttling — lag-based backpressure on metadata writes and the MCE consumer when the Metadata Change Log (MCL) pipeline falls behind. That mechanism is separate from HTTP rate limits; see MCP / Kafka ingest throttling.

What this is — and is not

DataHub has three separate load-protection mechanisms. They are easy to confuse because several can return 429. This guide covers GMS HTTP rate limiting in depth and MCP / Kafka ingest throttling in a dedicated section:

Mechanism	What it limits	When it applies	Configuration
GMS service rate limiting (this guide)	HTTP requests served by GMS — UI GraphQL, OpenAPI, Rest.li, `/auth/*` on GMS	Before/during request handling on the GMS pod	`RATE_LIMITS_*` env vars; bundled defaults under `datahub.gms.rateLimits` in `application.yaml`
MCP / Kafka ingest throttle (section below)	Sync metadata write APIs when MCL consumer lag is too high	Backpressure after ingest pipeline falls behind	`MCP_*` throttle env vars (see Environment Variables — MCP Throttle)
MCP consumer throttling (section below)	MCE consumer — pauses MCP consumption while MCL lag is high	Pipeline-side backpressure, not a per-client HTTP quota	`MCP_MCE_CONSUMER_THROTTLE_ENABLED`, `MCP_VERSIONED_`, `MCP_TIMESERIES_`

GMS HTTP rate limiting does not:

Rate-limit ingestion connectors (Python CLI, datahub ingest) — those are separate clients with their own retry behavior
Replace MCP throttle or Kafka lag backpressure — enabling GMS rate limits does not change metadataChangeProposal.throttle or consumer lag behavior
Apply to the Play frontend OAuth routes or an external WAF

Purpose (GMS service rate limiting only)

GMS rate limiting has two limit types with different semantics:

Capacity limits — Netflix Gradient2 adaptive in-flight caps: how many requests of a given class may be executing concurrently on a pod. Sheds serving load before latency and thread pools degrade.
Endpoint limits — Bucket4j token buckets: how many requests of a given path may be accepted per refill window (for example sign-up abuse guards on /auth/signUp).

How GMS service rate limiting differs from MCP / Kafka throttling

Both GMS service rate limiting and MCP ingest throttle can return 429 to API callers, but they answer different questions:

	GMS service rate limiting	MCP / Kafka ingest throttle
Question answered	“Is this GMS pod accepting more HTTP work right now?”	“Is the metadata pipeline too far behind to accept more writes?”
Trigger	Configured rules + live request latency (Gradient2) or token buckets	Kafka (or pgQueue) MCL topic backlog / lag
Retry-After	Capacity: `minRetryAfterSeconds`. Endpoint: `max(minRetryAfterSeconds, Bucket4j refill wait)` + jitter	Dynamic from exponential backoff on lag
Debug headers	`X-DataHub-RateLimit-*`	Same `X-DataHub-RateLimit-*` (`Type`: `ingest`)
Ops entry point	`/openapi/v1/rate-limits/`, `gms.rate_limit.` metrics	MCP throttle env vars / sensor logs and lag gauges

Scope

GMS HTTP serving path only: GraphQL, OpenAPI, Rest.li, and GMS /auth/* routes
Capacity limits are per-pod — no cluster-wide exact cap for adaptive in-flight in v1
Endpoint limits are cluster-wide (distributed via Hazelcast); see Endpoint limits

How it works

Each incoming request is evaluated against capacity limits and endpoint limits separately. The two types use different algorithms and do not share state.

Limit type	Question answered	Algorithm	Rules per request
Capacity	“How many requests like this may be in flight on this pod right now?”	Netflix Gradient2 (adaptive concurrency)	At most one — finest-grain match wins
Endpoint	“How many requests like this may be accepted in this time window?”	Bucket4j token bucket	At most one — finest-grain match wins

Cross-type behavior: A request must pass both checks when both match. For example, POST /auth/signUp typically acquires a slot from _default_capacity and consumes a token from an auth-signup endpoint rule. Denial on either type returns 429; if the endpoint bucket denies after capacity was acquired, the capacity slot is released.

Within a type, rules do not stack. A GraphQL search request matches either an operation-scoped rule in capacity.rules or the capacity.graphql pool — never both. Finer rules replace broader ones for matching traffic; they do not carve a sub-budget out of a parent pool.

flowchart LR
  REST[REST request] --> Filter[RateLimitFilter]
  Filter --> Acquire[Acquire capacity + endpoint]
  Acquire -->|allow| App[GMS handlers]
  Acquire -->|deny| R429[429 + headers]
  GQL[POST /api/graphql] --> Gate[GraphQLController gate]
  Gate --> AcquireGQL[Acquire after operation name known]
  AcquireGQL -->|allow| Execute[Async GraphQL execute]
  Execute --> Release[Release on future complete]

Capacity limits (adaptive in-flight)

Capacity limits cap concurrent in-flight requests per rule on each GMS pod. Each enabled capacity rule gets its own independent Gradient2 limiter. Limits adapt up or down based on observed latency for requests that acquired that rule's slot.

Independent pools — no shared budget. Capacity rules do not coordinate with each other. There is no parent/child accounting: an operation-scoped rule does not subtract from capacity.graphql, and neither subtracts from capacity.default. Each pool tracks its own in-flight count and adapts independently.

Because pools are disjoint, total concurrent load on a pod can exceed any single rule's maxLimit. With bundled defaults enabled, worst-case in-flight on one pod is roughly the sum of each active pool's current ceiling — for example _default_capacity (max 5000) + _graphql_capacity (max 2000) + any operation-scoped pools (each with its own max) can all be near their limits at the same time. All pools compete for the same underlying CPU, threads, and datastore, but the limiters do not see each other's pressure.

Planning guidance:

Treat capacity.default.maxLimit as the ceiling for Rest.li, OpenAPI, auth, and other non-GraphQL HTTP traffic.
Treat capacity.graphql.maxLimit as the ceiling for GraphQL operations not covered by a finer operation rule in capacity.rules.
Treat operation-scoped capacity rules as additional concurrent budgets for heavy queries — not as reductions to the GraphQL pool.
To enforce a hard total pod ceiling, lower capacity.default.maxLimit (and/or disable broader pools) rather than assuming finer rules inherit from a shared parent budget.
Capacity limits are per-pod. Cluster-wide concurrent load is approximately (GMS replica count) × (sum of per-pool inflight). There is no cluster-wide exact cap for adaptive capacity in v1.

Rule selection (finest grain wins within capacity type):

Rank	Match	Typical rule id
4	GraphQL operation name in `graphqlOperationNames`	`graphql-search-capacity`
3	GraphQL path (`capacity.graphql` or path rule without operations)	`_graphql_capacity`
2	Non-default path rule (for example `/auth/signUp`)	custom path rule
1	Global default (`capacity.default`)	`_default_capacity`

GraphQL lifecycle: The servlet filter does not acquire capacity for POST /api/graphql. The GraphQL controller acquires before execution and releases when the CompletableFuture completes. Gradient2 receives onSuccess only when the execution has no GraphQL errors (HTTP 200 with an errors array in the body is treated as a failed execution for adaptive tuning).

Operation name selection & threat model: Operation-scoped rule matching (rank 4) uses the request's operationName, or when it is omitted the first named operation in the query document. Both are client-controlled — the client authors the query and the JSON operationName, and the value is not cross-checked against the operation actually executed — so a client can rename or mis-name an operation to dodge or re-target an operation-scoped rule. This applies to both capacity.rules (adaptive, self-correcting) and endpoint.rules (hard token buckets): operation-name matching is advisory, not a security boundary. For abuse-resistant enforcement, key on dimensions the server observes rather than the client asserts — path/method (capacity.graphql, endpoint rules matched by pathPattern), the scoped actor chain (scoped:actor, keyed on the authenticated actor URN), or the heavy-resolver gate (scoped.heavyResolvers, keyed on the server-parsed top-level resolver names) — none of which the client can spoof. Note the unnamed-operation identity is also server-derived (sorted top-level field names). Operation-scoped rules are ignored entirely when capacity.graphql.operationRulesEnabled=false.

Async Spring MVC: For controllers that return CompletableFuture (for example /auth/*), the servlet filter registers a servlet AsyncListener and holds the capacity slot until async processing completes, times out, or errors — same effective lifecycle as synchronous handlers. GraphQL uses a dedicated controller gate instead of the filter.

Path class	Async?	Capacity tracking
`POST /api/graphql`	Yes (`CompletableFuture`)	Dedicated controller gate
Rest.li (`/entities/`, `/aspects/`, …)	Internal async, servlet blocks on `future.join()`	Filter chain waits for Rest.li
OpenAPI Spring controllers (`/openapi/**`)	Synchronous in current codebase	Servlet filter
Auth Spring controllers (`/auth/signUp`, …)	Yes — `CompletableFuture.supplyAsync`	Servlet filter + `AsyncListener`

Endpoint limits (token bucket)

Endpoint limits cap request rate over time using Bucket4j token buckets. Each endpoint rule has its own bucket; buckets are independent of capacity pools and of each other.

Combining with capacity: Endpoint rules answer a different question than capacity rules. When both match a request, both must allow — capacity checks in-flight concurrency, endpoint checks tokens per refill window. A path can be denied because its in-flight pool is full even when tokens remain, or because tokens are exhausted even when in-flight headroom exists.

Enable flags: Independent toggles per limiter type (both default to off):

Flag	Env var	Default	Effect
`capacity.enabled`	`RATE_LIMITS_CAPACITY_ENABLED`	`false`	Adaptive in-flight limits (Gradient2)
`endpoint.enabled`	`RATE_LIMITS_ENDPOINT_ENABLED`	`false`	Token-bucket limits (Bucket4j)

Enable one or both. Sub-pools (capacity.default.enabled, capacity.graphql.enabled) and typed rule lists (capacity.rules, endpoint.rules) further refine enforcement.

Cluster-wide enforcement (endpoint):

Endpoint limits always use Bucket4j with shared Hazelcast buckets when endpoint.enabled=true. Limits are cluster totals — capacity and refill* apply across all GMS replicas, not per pod.

Provision Hazelcast by setting endpoint.enabled=true (RATE_LIMITS_ENDPOINT_ENABLED). Startup fails if endpoint limits are enabled but Hazelcast cannot be reached. Bucket state is stored in rateLimits.endpoint.hazelcastMapName (default gmsRateLimitEndpointBuckets). Configure cluster connectivity via searchService.hazelcast.* / SEARCH_SERVICE_HAZELCAST_* in Environment Variables — Search.

Planning limits: Configure capacity / refill* as cluster-wide caps (e.g. 200 sign-ups/minute total across the fleet).

Example auth rule (under rateLimits.endpoint in policy files):

rateLimits:
  endpoint:
    enabled: true
    rules:
      - id: auth-signup
        pathPattern: /auth/signUp
        methods: [POST]
        capacity: 200
        refillTokens: 200
        refillPeriodSeconds: 60

429 response

{ "error": "Rate limit exceeded" }

Headers:

Header	Meaning
`X-DataHub-RateLimit-Rule`	Winning or denying rule id
`X-DataHub-RateLimit-Type`	`capacity`, `endpoint`, `ingest`, or `search`
`X-DataHub-RateLimit-Source`	`servlet-filter`, `graphql-gate`, `metadata-write`, or `opensearch`
`X-DataHub-RateLimit-Endpoint-Rule`	Endpoint rule id when both types applied on allow
`Retry-After`	Seconds (429 only)

Example:

curl -sv -X POST http://localhost:8080/auth/signUp -H 'Content-Type: application/json' -d '{}'

Enabling rate limiting

Default: off (capacity.enabled=false, endpoint.enabled=false).

Staging (env toggles only)

export RATE_LIMITS_CAPACITY_ENABLED=true
# or: export RATE_LIMITS_ENDPOINT_ENABLED=true
# restart GMS

Bundled defaults live under datahub.gms.rateLimits in application.yaml (alongside other GMS settings such as basePath and async).

Production checklist

Enable the limiter type(s) you need: RATE_LIMITS_CAPACITY_ENABLED=true and/or RATE_LIMITS_ENDPOINT_ENABLED=true
For endpoint caps: RATE_LIMITS_ENDPOINT_ENABLED=true (provisions Hazelcast automatically)
To override the bundled policy, mount a ConfigMap with your policy file and point RATE_LIMITS_CONFIG_FILE at it using a Spring resource prefix (e.g. file:/etc/datahub/rate-limits.yaml)
Rollout-restart GMS pods (config changes require restart in v1)
Verify (requires Manage System Operations privilege):
- GET /openapi/v1/rate-limits/config — effective merged config
- GET /openapi/v1/rate-limits/status — live limits on the pod that served the request
- Prometheus metrics gms.rate_limit.*

Per-actor rate limiting

Per-actor throttling is provided by the scoped chain's actor bucket (scoped:actor, keyed {tenantId}:actor:{urn}), enabled with scoped.enabled=true. It applies to both GraphQL and REST/OpenAPI/Rest.li (the servlet filter passes the authenticated actor). There is no bundled per-actor endpoint rule — endpoint.rules is empty by default, and there are no RATE_LIMITS_PER_ACTOR_* env vars.

Scoped chain — how each bucket is keyed

The scoped chain evaluates narrow → broad (actor → client class → global); the heavy-resolver gate runs separately, after the chain admits the request. What each bucket keys on:

Bucket	Key	Scope
`actor`	`{tenantId}:actor:{urn}`	Per authenticated actor. Always per-actor (its whole purpose); system principal exempt.
`browser` / `sdk`	`{tenantId}:{class}` — or `{tenantId}:{class}:{urn}`	Per class per tenant by default (a shared ceiling for all that class's traffic). With `scoped.<class>.perActor=true` it becomes per actor within the class. Set independently per class (`RATE_LIMITS_SCOPED_<CLASS>_PER_ACTOR`), and only applies when `clientClassEnabled=true`.
`global`	`global`	Fleet-wide, cross-tenant (un-prefixed, shared map). The absolute backstop.
`heavyResolvers.<f>`	`{tenantId}:op:{resolver}`	Per resolver per tenant — always, never per-actor. Keeping it tenant-wide prevents a tenant from multiplying a hot resolver's budget by spawning users; the `perActor` flag is ignored here.

Because global is the only cross-tenant bucket, size it well above the sum of all per-tenant traffic or it becomes the fleet-wide bottleneck. tenantId comes from RATE_LIMITS_TENANT_ID (the deployment's global.id in Helm); when unset, the tenant-scoped keys are un-prefixed (e.g. sdk, actor:{urn}), which is fine for a single deployment but collides on a shared store — set it in multi-deployment setups.

A lower-level perActor endpoint-rule mechanism still exists for advanced, manual per-path use: setting perActor: true on an endpoint (token-bucket) rule makes GMS keep a separate Bucket4j bucket per authenticated actor URN rather than one shared bucket. You author such a rule yourself (in a mounted config file); nothing enables it by default. Prefer the scoped actor bucket for ordinary per-actor GraphQL throttling.

Constraints (for a hand-authored perActor endpoint rule):

perActor is supported only on endpoint.rules[]. Setting it on a capacity.rules[] entry causes GMS to refuse to start with IllegalStateException.
Per-actor sharding applies to the GraphQL POST path only (/api/graphql by default). A perActor rule on a non-GraphQL path logs a warning at startup and falls back to pass-through (no per-actor bucket is consulted).
perActor requires endpoint.enabled=true (Hazelcast).

System actor exemption:

The internal system principal (actor URN matching DATAHUB_SYSTEM_CLIENT_ID, default urn:li:corpuser:__datahub_system) is resolved to null before the rate limit engine is called. A perActor rule with a null actor URN skips the per-actor consume entirely — the request proceeds without consuming a token. This means high-volume internal GMS operations are never throttled by per-actor rules.

Bucket lifecycle in Hazelcast:

Per-actor bucket entries are stored under a composite Hazelcast key {ruleId}:actor:{actorUrn}. Entries are evicted after bucketMaxIdleSeconds of inactivity (default 300 s, RATE_LIMITS_ENDPOINT_BUCKET_MAX_IDLE_SECONDS). An LRU size cap of bucketMaxSize entries per node prevents unbounded Hazelcast memory growth (default 100 000, RATE_LIMITS_ENDPOINT_BUCKET_MAX_SIZE).

Merge behavior when overriding via a mounted config file:

Bundled defaults live in application.yaml. A file mounted at RATE_LIMITS_CONFIG_FILE is loaded by Spring as a property source and layered on top (it must use a Spring resource prefix, e.g. file:/etc/datahub/rate-limits.yaml). Because this is Spring property binding:

Scalars and map entries (e.g. the scoped bucket sizes, scoped.heavyResolvers.*) from the mounted file override or add to the bundled values key by key.
Rule lists (capacity.rules, endpoint.rules) are bound by index, not replaced wholesale — keep rule lists defined in a single source. application.yaml ships empty rule lists, so a mounted file that declares rules simply provides them.

datahub:
  gms:
    rateLimits:
      endpoint:
        rules:
          - id: auth-signup
            pathPattern: /auth/signUp
            methods: [POST]
            capacity: 200
            refillTokens: 200
            refillPeriodSeconds: 60

Observability for per-actor rules:

The gms.rate_limit.endpoint.remaining metric is not registered for perActor rules. The ruleId-keyed shared bucket is never consumed by per-actor requests, so its gauge would always read full and mislead operators. Monitor per-actor denials via gms.rate_limit.requests{outcome=deny,rule_id=<your-rule-id>} instead.

Example — hand-authored per-actor GraphQL throttle (mounted config file; nothing like this ships by default — prefer scoped:actor):

datahub:
  gms:
    rateLimits:
      endpoint:
        enabled: true
        rules:
          - id: graphql-per-actor
            enabled: true
            pathPattern: /api/graphql
            methods: [POST]
            perActor: true
            capacity: 500
            refillTokens: 500
            refillPeriodSeconds: 60

Scope of per-actor limiting: The scoped actor bucket (scoped:actor) applies to both GraphQL and REST/OpenAPI/Rest.li — the servlet filter passes the authenticated actor, so a noisy SDK principal on REST paths is isolated the same way it is on GraphQL (the system principal and unauthenticated requests are exempt). The browser-vs-SDK client-class distinction (clientClassEnabled, scoped:browser/scoped:sdk, modeled as ClientClass) also applies across both. By default the class buckets are one shared ceiling per class per tenant, but each class can be flipped to per-actor independently via scoped.<class>.perActor (RATE_LIMITS_SCOPED_<CLASS>_PER_ACTOR) — e.g. give every SDK caller its own tight sdk budget ({tenant}:sdk:{urn}) while keeping browser a shared ceiling. The lower-level perActor flag on endpoint rules remains GraphQL-POST-only (see the per-actor endpoint-rule constraints above).

Configuration reference

Bundled defaults live entirely in application.yaml (every value env-overridable). No per-actor endpoint rule ships; endpoint.rules is empty:

rateLimits:
  failOpen: true
  minRetryAfterSeconds: 60
  retryAfterJitterPercent: 10
  excludedPaths: /health,/health/live,/actuator/prometheus,/openapi/v1/rate-limits/**
  capacity:
    enabled: false # RATE_LIMITS_CAPACITY_ENABLED
    default:
      enabled: true # RATE_LIMITS_CAPACITY_DEFAULT_ENABLED
      initialLimit: 200 # RATE_LIMITS_CAPACITY_DEFAULT_INITIAL_LIMIT
      minLimit: 20 # RATE_LIMITS_CAPACITY_DEFAULT_MIN_LIMIT
      maxLimit: 5000 # RATE_LIMITS_CAPACITY_DEFAULT_MAX_LIMIT
    graphql:
      enabled: true # RATE_LIMITS_CAPACITY_GRAPHQL_ENABLED
      pathPattern: /api/graphql # RATE_LIMITS_CAPACITY_GRAPHQL_PATH_PATTERN
      operationRulesEnabled: true # RATE_LIMITS_CAPACITY_GRAPHQL_OPERATION_RULES_ENABLED
      initialLimit: 100 # RATE_LIMITS_CAPACITY_GRAPHQL_INITIAL_LIMIT
      minLimit: 20 # RATE_LIMITS_CAPACITY_GRAPHQL_MIN_LIMIT
      maxLimit: 2000 # RATE_LIMITS_CAPACITY_GRAPHQL_MAX_LIMIT
    rules: []
  endpoint:
    enabled: false # RATE_LIMITS_ENDPOINT_ENABLED
    hazelcastMapName: gmsRateLimitEndpointBuckets # RATE_LIMITS_ENDPOINT_HAZELCAST_MAP
    bucketMaxIdleSeconds: 300 # RATE_LIMITS_ENDPOINT_BUCKET_MAX_IDLE_SECONDS
    bucketMaxSize: 100000 # RATE_LIMITS_ENDPOINT_BUCKET_MAX_SIZE
    rules: []
  scoped:
    enabled: false # RATE_LIMITS_SCOPED_ENABLED
    refundDisabled: false # RATE_LIMITS_SCOPED_REFUND_DISABLED
    # Bucket sizes (env-overridable per deployment). Each field has its own env var:
    # RATE_LIMITS_SCOPED_<BUCKET>_{CAPACITY,REFILL_TOKENS,REFILL_PERIOD_SECONDS}.
    # refillTokens defaults to that bucket's CAPACITY; refillPeriodSeconds defaults to 60.
    # perActor (browser/sdk only): true → key that class bucket per actor; false (default) → one
    # shared ceiling per class per tenant. RATE_LIMITS_SCOPED_<CLASS>_PER_ACTOR.
    actor: { capacity: 2000, refillTokens: 2000, refillPeriodSeconds: 60 }
    browser:
      {
        perActor: false,
        capacity: 5000,
        refillTokens: 5000,
        refillPeriodSeconds: 60,
      }
    sdk:
      {
        perActor: false,
        capacity: 500,
        refillTokens: 500,
        refillPeriodSeconds: 60,
      }
    global: { capacity: 20000, refillTokens: 20000, refillPeriodSeconds: 60 }
    heavyResolvers: {} # per-resolver buckets, always per-tenant; add via mounted file
  metrics:
    detailed: false

Tier 1 — environment toggles

Key environment variables (full list at Environment Variables — GMS Rate Limiting):

Environment variable	Default	Description
`RATE_LIMITS_CAPACITY_ENABLED`	`false`	Enable adaptive in-flight (Gradient2) capacity limiting
`RATE_LIMITS_ENDPOINT_ENABLED`	`false`	Enable token-bucket endpoint limiting (requires Hazelcast)
`RATE_LIMITS_FAIL_OPEN`	`true`	Allow requests when evaluation throws an unexpected error
`RATE_LIMITS_MIN_RETRY_AFTER`	`60`	Minimum `Retry-After` seconds on 429 responses
`RATE_LIMITS_RETRY_AFTER_JITTER_PERCENT`	`10`	Jitter percentage added to `Retry-After`
`RATE_LIMITS_EXCLUDED_PATHS`	`/health,/health/live,...`	Comma-separated Ant paths excluded from all rate limiting
`RATE_LIMITS_CONFIG_FILE`	(unset)	Spring resource URI of an override policy file, layered on the bundled defaults — must include a prefix (e.g. `file:/etc/datahub/rate-limits.yaml`)
`RATE_LIMITS_METRICS_DETAILED`	`false`	Enable detailed per-rule metric tags
`RATE_LIMITS_CAPACITY_DEFAULT_ENABLED`	`true`	Enable `_default_capacity` pool
`RATE_LIMITS_CAPACITY_DEFAULT_INITIAL_LIMIT`	`200`	Gradient2 starting limit for default pool
`RATE_LIMITS_CAPACITY_DEFAULT_MIN_LIMIT`	`20`	Gradient2 floor for default pool
`RATE_LIMITS_CAPACITY_DEFAULT_MAX_LIMIT`	`5000`	Gradient2 ceiling for default pool
`RATE_LIMITS_CAPACITY_GRAPHQL_ENABLED`	`true`	Enable `_graphql_capacity` pool
`RATE_LIMITS_CAPACITY_GRAPHQL_PATH_PATTERN`	`/api/graphql`	Path matched by the GraphQL capacity pool
`RATE_LIMITS_CAPACITY_GRAPHQL_OPERATION_RULES_ENABLED`	`true`	Allow per-operation capacity rules for GraphQL
`RATE_LIMITS_CAPACITY_GRAPHQL_INITIAL_LIMIT`	`100`	Gradient2 starting limit for GraphQL pool
`RATE_LIMITS_CAPACITY_GRAPHQL_MIN_LIMIT`	`20`	Gradient2 floor for GraphQL pool
`RATE_LIMITS_CAPACITY_GRAPHQL_MAX_LIMIT`	`2000`	Gradient2 ceiling for GraphQL pool
`RATE_LIMITS_ENDPOINT_HAZELCAST_MAP`	`gmsRateLimitEndpointBuckets`	Hazelcast map name for endpoint buckets
`RATE_LIMITS_ENDPOINT_BUCKET_MAX_IDLE_SECONDS`	`300`	Idle eviction window (seconds) for per-actor bucket entries
`RATE_LIMITS_ENDPOINT_BUCKET_MAX_SIZE`	`100000`	LRU size cap for endpoint bucket entries per Hazelcast node
`RATE_LIMITS_SCOPED_ENABLED`	`false`	Enable the scoped chain (per-actor → class → global)
`RATE_LIMITS_SCOPED_ACTOR_CAPACITY`	`2000`	Per-actor (`scoped:actor`) bucket size — the per-actor GraphQL throttle
`RATE_LIMITS_SCOPED_SDK_CAPACITY`	`500`	SDK/non-browser class (`scoped:sdk`) bucket size
`RATE_LIMITS_SCOPED_BROWSER_CAPACITY`	`5000`	Browser class (`scoped:browser`) bucket size
`RATE_LIMITS_SCOPED_GLOBAL_CAPACITY`	`20000`	Fleet-wide (`scoped:global`) ceiling
`RATE_LIMITS_SCOPED_<BUCKET>_REFILL_TOKENS`	(= bucket capacity)	Tokens refilled per period for `<BUCKET>` (`ACTOR`/`BROWSER`/`SDK`/`GLOBAL`); defaults to that bucket's capacity
`RATE_LIMITS_SCOPED_<BUCKET>_REFILL_PERIOD_SECONDS`	`60`	Refill period (seconds) for `<BUCKET>` (`ACTOR`/`BROWSER`/`SDK`/`GLOBAL`)
`RATE_LIMITS_SCOPED_<BUCKET>_PER_ACTOR`	`false`	`BROWSER`/`SDK` only: `true` keys that class bucket per actor (`{tenant}:{class}:{urn}`); `false` = one shared ceiling per class per tenant. Set independently per class. Ignored for `ACTOR` (already per-actor) / `GLOBAL` (fleet).

Tier 2 — override policy file

Bundled defaults ship in application.yaml (Tier 1). To override them per deployment, mount your own file and point RATE_LIMITS_CONFIG_FILE at it with a Spring resource prefix (e.g. file:/etc/datahub/rate-limits.yaml); it is loaded as a property source and layered on top. When RATE_LIMITS_CONFIG_FILE is unset it resolves to the bundled rate-limit-config.yaml, which is empty and contributes nothing. Override files use the full datahub.gms.rateLimits: path (the same keys Spring binds), not a bare rateLimits: fragment.

Merge behavior: scalars and map entries override/add key by key; rule lists are index-bound, so define each rule list in a single source (application.yaml ships empty rule lists). Most per-tenant tuning is done with RATE_LIMITS_SCOPED_* env vars rather than a file — the override file is mainly for the scoped.heavyResolvers map and rule lists, which a scalar env var can't express (and which are empty in the bundled defaults, so a mounted file simply provides them).

How it loads (and what changed): both the bundled defaults and the mounted override are loaded as ordinary Spring @PropertySources and bound to RateLimitProperties via relaxed @ConfigurationProperties binding — the same path application.yaml already uses. There is no bespoke config loader. This is intentionally simpler than an earlier design that used a custom Jackson loader, and it means:

The mounted override is enabled just by pointing RATE_LIMITS_CONFIG_FILE at a file (presence = enabled). There is no separate RATE_LIMITS_CONFIG_FILE_ENABLED flag.
There is no inline-JSON overlay env var (RATE_LIMITS_CONFIG_JSON). Put overrides in the mounted YAML file instead.
${ENV:default} placeholders resolve uniformly, and OS environment variables outrank both files — so RATE_LIMITS_* env vars always win, which is the recommended way to tune scalars per deployment.

RATE_LIMITS_CONFIG_FILE_ENABLED and RATE_LIMITS_CONFIG_JSON are no longer honored; GMS logs a startup WARN if either is set so the change isn't silent. Benefits of the @PropertySource approach: one well-understood binding/precedence model (identical to the rest of GMS config), no custom merge code to maintain, and env-var overrides that reliably take precedence.

datahub:
  gms:
    rateLimits:
      capacity:
        rules:
          - id: graphql-search-capacity
            pathPattern: /api/graphql
            methods: [POST]
            graphqlOperationNames: [searchAcrossEntities, scrollAcrossEntities]
            initialLimit: 30
            maxLimit: 400
      endpoint:
        rules:
          - id: auth-signup
            pathPattern: /auth/signUp
            methods: [POST]
            capacity: 200
            refillTokens: 200
            refillPeriodSeconds: 60
      scoped:
        heavyResolvers:
          searchAcrossEntities:
            { capacity: 100, refillTokens: 100, refillPeriodSeconds: 60 }

Helm-style wiring:

env:
  - name: RATE_LIMITS_ENDPOINT_ENABLED
    value: "true"
  - name: RATE_LIMITS_CAPACITY_ENABLED
    value: "true"
  # Spring resource URI — note the file: prefix.
  - name: RATE_LIMITS_CONFIG_FILE
    value: file:/etc/datahub/rate-limits/rate-limits.yaml
volumeMounts:
  - name: rate-limits-config
    mountPath: /etc/datahub/rate-limits
    readOnly: true
volumes:
  - name: rate-limits-config
    configMap:
      name: datahub-rate-limits

Base path

Author logical paths in config (/api/graphql, /auth/signUp). GMS strips the configured base path at runtime. See Base Path Configuration.

Rule types (quick reference)

Config source	Type	Semantics	Example
`capacity.default`	Capacity	Global adaptive in-flight per pod	Rest.li, OpenAPI, auth fallback
`capacity.graphql`	Capacity	GraphQL POST adaptive in-flight per pod	UI load ceiling
`capacity.rules[]`	Capacity	Finer adaptive in-flight pool (replaces broader match)	`searchAcrossEntities`
`endpoint.rules[]` (`perActor: false`)	Endpoint	Shared token-bucket rate per refill window (cluster-wide)	`/auth/signUp`, `batchIngest`
`scoped:actor`	Scoped	Per-actor token-bucket; each authenticated actor has an independent quota	per-actor GraphQL throttle (default model)
`endpoint.rules[]` (`perActor: true`)	Endpoint	Per-actor token-bucket on a hand-authored rule (advanced; none ships)	manual per-path rule

See Capacity limits (adaptive in-flight) and Endpoint limits (token bucket) for selection, pooling, and planning details.

Observability

Prometheus metrics (tagged by rule_id, type, outcome, and optionally graphql_operation):

Metric	Description
`gms.rate_limit.requests`	Allow/deny counts
`gms.rate_limit.adaptive.limit`	Current Gradient2 ceiling
`gms.rate_limit.adaptive.inflight`	In-flight gauge
`gms.rate_limit.endpoint.remaining`	Token bucket headroom
`gms.rate_limit.fail_open`	Fail-open events (eval threw, request allowed because `failOpen=true`), tagged by `stage`

gms.rate_limit.requests sets graphql_operation to the resolved operation name only when an operation-scoped rule matched (a rule with graphqlOperationNames); otherwise none. This bounds metric cardinality — arbitrary client-supplied operation names on the general GraphQL pool are not tagged.

Sampling — important for reading outcome=allow: to keep hot-path overhead low, allowed requests are recorded at a 1-in-100 sample rate, while denied requests are always recorded. So gms.rate_limit.requests{outcome=allow} undercounts actual allowed traffic by ~100× — multiply by 100 for an approximate volume, and never compute an exact allow/deny ratio from the raw counters. The ×100 estimate is a Poisson sample and has high relative variance at low volumes: a handful of sampled events implies a wide confidence interval (e.g. ~10 sampled ≈ 1,000 ± ~300), so treat low-traffic allow counts as rough order-of-magnitude only and prefer longer windows or metrics.detailed when you need accuracy. Deny-based alerting is unaffected (denials are exact). Set RATE_LIMITS_METRICS_DETAILED=true (metrics.detailed) to disable sampling and record every request (higher cardinality/overhead — use selectively).

Suggested alerts: sustained outcome=deny, adaptive limit pinned at minLimit, endpoint remaining near zero on auth rules. Alert on any sustained gms.rate_limit.fail_open rate — fail-open means rate-limit evaluation is throwing (e.g. Hazelcast connectivity loss) and requests are passing unlimited, so the protection is effectively off until it clears; there is no circuit-breaker, so this metric is the signal to investigate. For capacity planning, sum gms.rate_limit.adaptive.inflight across rule_id tags on a pod — each tag is an independent pool.

Inspection API (Manage System Operations privilege):

Endpoint	Purpose
`GET /openapi/v1/rate-limits/config`	Effective merged configuration
`GET /openapi/v1/rate-limits/status`	Live state on the pod that served the request

Status response includes capacityEnabled, endpointEnabled, scopedEnabled, plus per-rule adaptive (limit/inflight) and endpoint (remaining/capacity) maps, and a scoped map with the fixed-key buckets (global, browser, sdk — each remaining/capacity). Per-actor scoped buckets are omitted since their keys are unbounded.

MCP / Kafka ingest throttling

Lag-based backpressure for the metadata write pipeline. When the Metadata Change Log (MCL) consumer falls behind, DataHub can slow or reject new writes so the backlog can drain — instead of letting Kafka lag grow without bound.

This is not GMS HTTP rate limiting (RATE_LIMITS_*). It is configured under metadataChangeProposal.throttle and uses MCP_* environment variables. Full env-var tables: Environment Variables — Metadata Change Proposal.

What it protects

Metadata writes flow roughly:

API / connector  →  MCP (sync write or async proposal)  →  primary store + MCL
                                                              ↓
                                                    MAE consumer (search, graphs, …)

If MAE (or equivalent) cannot keep up with MCL production, median consumer lag rises. The throttle sensor watches that lag and, when thresholds are exceeded, applies backpressure to configured components.

Component	Env var	Default	Effect when lag is high
API requests (`components.apiRequests`)	`MCP_API_REQUESTS_THROTTLE_ENABLED`	`false`	Sync metadata write APIs throw `APIThrottleException` → HTTP 429 with `X-DataHub-RateLimit-Type: ingest`
MCE consumer (`components.mceConsumer`)	`MCP_MCE_CONSUMER_THROTTLE_ENABLED`	`false`	Pauses the MCP Kafka listener container so async proposals stop being consumed until lag clears

You can enable either or both. Lag monitoring itself is controlled separately per MCL topic family (versioned vs timeseries).

How lag is measured

A scheduled sensor (KafkaThrottleSensor, or PgQueueThrottleSensor when datahub.messaging.transport=pgqueue) runs every MCP_THROTTLE_UPDATE_INTERVAL_MS (default 60s):

Reads committed offsets for the MCL consumer group (default generic-mae-consumer-job-client, overridable via METADATA_CHANGE_LOG_KAFKA_CONSUMER_GROUP_ID)
Compares them to topic end offsets for the versioned and timeseries MCL topics
Computes median lag per topic family
If median lag exceeds the configured threshold and that topic family is enabled, enters a throttled state with exponential backoff

flowchart TD
  Sensor[Throttle sensor every updateIntervalMs] --> Lag[Median MCL lag]
  Lag -->|lag > threshold| Backoff[Exponential backoff]
  Backoff --> API[API requests component: 429 on sync writes]
  Backoff --> MCE[MCE consumer component: pause listener]
  Lag -->|lag ≤ threshold| Clear[Clear throttle / resume]

Sensor requirements:

MCP_THROTTLE_UPDATE_INTERVAL_MS must be > 0 (default 60000). Set to 0 to disable the sensor entirely (NoOpSensor).
At least one of MCP_VERSIONED_THROTTLE_ENABLED / MCP_TIMESERIES_THROTTLE_ENABLED must be true for lag checks to run.
Kafka bootstrap must be resolvable (or pgQueue store available for the pgQueue transport).

Versioned vs timeseries lag

Versioned and timeseries MCL topics are throttled independently:

Topic family	Enable flag	Threshold (default)	Applied to
Versioned	`MCP_VERSIONED_THROTTLE_ENABLED`	`MCP_VERSIONED_THRESHOLD` (`4000`)	Non-timeseries sync writes; MCE pause callbacks
Timeseries	`MCP_TIMESERIES_THROTTLE_ENABLED`	`MCP_TIMESERIES_THRESHOLD` (`4000`)	Timeseries sync writes; MCE pause callbacks

Backoff tuning (same shape for both families):

Setting	Versioned env var	Timeseries env var	Default
Max attempts	`MCP_VERSIONED_MAX_ATTEMPTS`	`MCP_TIMESERIES_MAX_ATTEMPTS`	`1000`
Initial interval	`MCP_VERSIONED_INITIAL_INTERVAL_MS`	`MCP_TIMESERIES_INITIAL_INTERVAL_MS`	`100`
Multiplier	`MCP_VERSIONED_MULTIPLIER`	`MCP_TIMESERIES_MULTIPLIER`	`10`
Max interval	`MCP_VERSIONED_MAX_INTERVAL_MS`	`MCP_TIMESERIES_MAX_INTERVAL_MS`	`30000`

While throttled, the sensor sleeps for the current backoff interval before notifying resume callbacks. Suggested wait is surfaced to API clients as Retry-After (seconds) on 429 responses.

API request throttling behavior

When MCP_API_REQUESTS_THROTTLE_ENABLED=true and the sensor reports active lag:

Sync metadata writes (Rest.li / OpenAPI / GraphQL entity updates that commit to the primary store) are evaluated by APIThrottle.
Async non-timeseries writes (proposal written to the MCP topic only) are not rejected here — use MCE consumer throttling to slow that path.
Browser UI traffic (AgentClass.BROWSER) is exempt so interactive UI edits are not blocked by ingest backlog. SDK / CLI / connector traffic is not exempt.
Internal calls without a request context are exempt.

429 response (OpenAPI / Rest.li handlers convert APIThrottleException):

{ "error": "Too many requests" }

Headers (same family as GMS rate limits):

Header	Typical value
`X-DataHub-RateLimit-Type`	`ingest`
`X-DataHub-RateLimit-Source`	`metadata-write`
`X-DataHub-RateLimit-Rule`	Active types, e.g. `MCL_VERSIONED_LAG`
`Retry-After`	Suggested wait in seconds (when backoff is known)

MCE consumer throttling behavior

When MCP_MCE_CONSUMER_THROTTLE_ENABLED=true, the MCE consumer registers a callback on the same lag sensor. On throttle it pauses the MCP listener container; after the backoff sleep it resumes. That slows async ingest without returning 429 to the original API caller (the caller already returned after writing the proposal).

Apply the same MCP_VERSIONED_* / MCP_TIMESERIES_* flags and thresholds on GMS and MCE Consumer so both see consistent lag policy. See Environment Variables — MCP for the component list.

Enabling (typical staging / production)

Default: all MCP throttle flags are off.

Minimum useful enablement for API backpressure on versioned lag:

export MCP_API_REQUESTS_THROTTLE_ENABLED=true
export MCP_VERSIONED_THROTTLE_ENABLED=true
# optional: also pause async MCP consumption
# export MCP_MCE_CONSUMER_THROTTLE_ENABLED=true
# restart GMS (and MCE consumer if mceConsumer throttle is enabled)

Tune threshold to your cluster’s normal MCL lag headroom. 4000 is a starting point, not a universal safe value — set it above expected steady-state median lag and below the lag that causes search/graph staleness incidents.

Config lives under metadataChangeProposal.throttle in application.yaml:

metadataChangeProposal:
  throttle:
    updateIntervalMs: ${MCP_THROTTLE_UPDATE_INTERVAL_MS:60000}
    components:
      mceConsumer:
        enabled: ${MCP_MCE_CONSUMER_THROTTLE_ENABLED:false}
      apiRequests:
        enabled: ${MCP_API_REQUESTS_THROTTLE_ENABLED:false}
    versioned:
      enabled: ${MCP_VERSIONED_THROTTLE_ENABLED:false}
      threshold: ${MCP_VERSIONED_THRESHOLD:4000}
      maxAttempts: ${MCP_VERSIONED_MAX_ATTEMPTS:1000}
      initialIntervalMs: ${MCP_VERSIONED_INITIAL_INTERVAL_MS:100}
      multiplier: ${MCP_VERSIONED_MULTIPLIER:10}
      maxIntervalMs: ${MCP_VERSIONED_MAX_INTERVAL_MS:30000}
    timeseries:
      enabled: ${MCP_TIMESERIES_THROTTLE_ENABLED:false}
      threshold: ${MCP_TIMESERIES_THRESHOLD:4000}
      # … same backoff shape as versioned

Observability

Sensor logs median MCL lag on each refresh and warns when entering throttle (Throttled Topic: … Duration: … MedianLag: …).
Gauges / counters on the sensor class: {topic}_throttled (0/1) and {topic}_throttledCount.
API denials share the X-DataHub-RateLimit-* headers with Type=ingest — distinguish from capacity/endpoint denials (Type=capacity / endpoint) described earlier in this guide.