datadog.md

Description

The datadog Plugin supports integration with Datadog, one of the most widely used observability services for cloud applications. When enabled, the Plugin pushes metrics to the DogStatsD server, which comes bundled with the Datadog agent, over UDP protocol.

This Plugin provides the ability to push metrics as a batch to the external Datadog agent over UDP. It might take some time to receive the metric data. It will be automatically sent after the timer function in the batch processor expires.

Attributes

Name	Type	Required	Default	Valid values	Description
prefer_name	boolean	False	true	[true, false]	If true, exports Route/Service name instead of their ID in metric tags.
include_path	boolean	False	false	[true, false]	If true, includes the path pattern in metric tags.
include_method	boolean	False	false	[true, false]	If true, includes the HTTP method in metric tags.
constant_tags	array	False	[]		Static key-value tags attached to all metrics generated by this Route. Useful for grouping metrics over certain signals.
batch_max_size	integer	False	1000	[1,...]	Maximum number of metric entries per batch. Once reached, the batch is sent to the Datadog agent. Set to 1 for immediate processing.
inactive_timeout	integer	False	5	[1,...]	Maximum time in seconds to wait for new metric entries before sending the batch. The value should be smaller than buffer_duration.
buffer_duration	integer	False	60	[1,...]	Maximum time in seconds from the earliest metric entry allowed before sending the batch.
retry_delay	integer	False	1	[0,...]	Time interval in seconds to retry sending the batch if the previous attempt failed.
max_retry_count	integer	False	0	[0,...]	Maximum number of unsuccessful retries before dropping the metric entries.

This Plugin supports using batch processors to aggregate and process metric data in a batch. This avoids the need to submit data too frequently. The batch processor submits data every 5 seconds or when the data in the queue reaches 1000. See Batch Processor for more information or to customize the configuration.

Plugin Metadata

You can configure the Plugin through Plugin metadata.

Name	Type	Required	Default	Description
host	string	False	"127.0.0.1"	DogStatsD server host address.
port	integer	False	8125	DogStatsD server port.
namespace	string	False	"apisix"	Prefix for all custom metrics sent by the APISIX agent. Useful for finding entities for metrics graph. For example, apisix.request.counter.
constant_tags	array	False	[ "source:apisix" ]	Static tags to embed into generated metrics. Useful for grouping metrics over certain signals. See defining tags for more.

Metrics

The Plugin exports the following metrics by default.

All metrics will be prefixed by the namespace configured in metadata. For example, if namespace is set to apisix, the request.counter metric will be exported as apisix.request.counter in Datadog.

Name	StatsD Type	Description
request.counter	Counter	Number of requests received.
request.latency	Histogram	Time taken to process the request, in milliseconds.
upstream.latency	Histogram	Time taken to proxy the request to the upstream server until a response is received, in milliseconds.
apisix.latency	Histogram	Time taken by the APISIX agent to process the request, in milliseconds.
ingress.size	Timer	Request body size in bytes.
egress.size	Timer	Response body size in bytes.

Tags

The Plugin exports metrics with the following tags. When there are no suitable values for any particular tag, the tag will be omitted.

Name	Description
route_name	Name of the Route. If not present or if the attribute prefer_name is set to false, falls back to the Route ID.
service_name	Name of the Service. If not present or if the attribute prefer_name is set to false, falls back to the Service ID.
consumer	Username of the Consumer if the Route is connected to a Consumer.
balancer_ip	IP address of the upstream balancer that processes the current request.
response_status	HTTP response status code, such as 201, 404, or 503.
response_status_class	HTTP response status code class, such as 2xx, 4xx, or 5xx. Available from APISIX version 3.14.0.
scheme	Request scheme, such as HTTP and gRPC.
path	HTTP path pattern. Only available if the attribute include_path is set to true. Available from APISIX version 3.14.0.
method	HTTP method. Only available if the attribute include_method is set to true. Available from APISIX version 3.14.0.

Examples

The following examples demonstrate how you can configure the datadog Plugin for different scenarios.

Before proceeding, make sure you have installed the Datadog agent, which collects events and metrics from monitored objects and sends them to Datadog.

Start the Datadog agent with your API key, Datadog site, and hostname. Set DD_DOGSTATSD_NON_LOCAL_TRAFFIC to true to listen to DogStatsD packets from other containers:

docker run -d \
  --name dogstatsd-agent \
  -e DD_API_KEY=<your-api-key> \
  -e DD_SITE="us5.datadoghq.com" \
  -e DD_HOSTNAME=apisix.quickstart \
  -e DD_DOGSTATSD_NON_LOCAL_TRAFFIC=true \
  -p 8125:8125/udp \
  datadog/dogstatsd:latest

You can configure most options in the agent's main configuration file datadog.yaml through environment variables prefixed with DD_. For more information, see agent environment variables.

:::note

You can fetch the admin_key from config.yaml and save to an environment variable with the following command:

admin_key=$(yq '.deployment.admin.admin_key[0].key' conf/config.yaml | sed 's/"//g')

:::

Update Datadog Agent Address and Metadata

By default, the Plugin expects the DogStatsD server to be available at 127.0.0.1:8125. To customize the address and other metadata, update the Plugin metadata. Set the host to your Datadog agent address, the port to the agent listening port, the namespace to prefix all metrics, and add any constant tags:

curl "http://127.0.0.1:9180/apisix/admin/plugin_metadata/datadog" -X PUT \
  -H "X-API-KEY: ${admin_key}" \
  -d '{
    "host": "192.168.0.90",
    "port": 8125,
    "namespace": "apisix",
    "constant_tags": [
      "source:apisix",
      "service:custom"
    ]
  }'

To reset to default configuration, send a request to the datadog Plugin metadata with an empty body:

curl "http://127.0.0.1:9180/apisix/admin/plugin_metadata/datadog" -X PUT \
  -H "X-API-KEY: ${admin_key}" \
  -d '{}'

Monitor Route Metrics

The following example demonstrates how to send the metrics of a particular Route to Datadog.

Create a Route with the datadog Plugin. Set batch_max_size to 1 to send each metric immediately, and max_retry_count to 0 to disallow retries if metrics were unsuccessfully sent:

curl "http://127.0.0.1:9180/apisix/admin/routes" -X PUT \
  -H "X-API-KEY: ${admin_key}" \
  -d '{
    "id": "datadog-route",
    "uri": "/anything",
    "plugins": {
      "datadog": {
        "batch_max_size": 1,
        "max_retry_count": 0
      }
    },
    "upstream": {
      "type": "roundrobin",
      "nodes": {
        "httpbin.org:80": 1
      }
    }
  }'

Generate a few requests to the Route:

curl "http://127.0.0.1:9080/anything"

In Datadog, select Metrics from the left menu and go to Explorer. Select apisix.ingress.size.count as the metric. You should see the count reflecting the number of requests generated.