Cardinality limit plugin

Limits the cardinality of fields on events, drops events or just do nothing.

Examples

Discarding events with high cardinality field:

pipelines:
  example_pipeline:
    ...
    - type: cardinality
      limit: 2
      action: discard
      ttl: 1m
      metric_prefix: service_client
      key:
        - service
      fields:
        - client_id
    ...

Events:

{"service": "registration", "client_id": "1"}
{"service": "registration", "client_id": "1"}
{"service": "registration", "client_id": "2"}
{"service": "registration", "client_id": "3"} // will be discarded

---

Remove high cardinality fields:

pipelines:
  example_pipeline:
    ...
    - type: cardinality
      limit: 2
      action: remove_fields
      ttl: 1m
      metric_prefix: service_client
      key:
        - service
      fields:
        - client_id
    ...

The original events:

{"service": "registration", "client_id": "1"}
{"service": "registration", "client_id": "2"}
{"service": "registration", "client_id": "3"}

The resulting events:

{"service": "registration", "client_id": "1"}
{"service": "registration", "client_id": "2"}
{"service": "registration"}

Config params

key []cfg.FieldSelector required

Fields used to group events before calculating cardinality. Events with the same key values are aggregated together. Required for proper cardinality tracking per logical group.

fields []cfg.FieldSelector required

Target fields whose unique values are counted within each key group. The plugin monitors how many distinct values these fields contain. Required to define what constitutes high cardinality.

action string default=nothing options=discard|remove_fields|nothing

Action to perform when cardinality limit is exceeded. Determines whether to discard events, remove fields, or just monitor. Choose based on whether you need to preserve other event data.

metric_prefix string

Prefix added to metric names for better organization. Useful when running multiple instances to avoid metric name collisions. Leave empty for default metric naming.

limit int default=10000

Maximum allowed number of unique values for monitored fields. When exceeded within a key group, the configured action triggers. Set based on expected diversity and system capacity.

ttl cfg.Duration default=1h

Time-to-live for cardinality tracking cache entries. Prevents unbounded memory growth by forgetting old unique values. Should align with typical patterns of field value changes.

Generated using insane-doc