Client IP based Rate Limiting

Rate limits in HAProxy are based on stick-tables. The concept of stick-tables is explained in this blog article. It covers all relevant parts and gives a general idea on how one could rate limit based on certain attributes.

HAProxy-boshrelease can be configured to enforce these rate limits based on requests and connections per second per IP.

Configuration Options

See also jobs/haproxy/spec:

There are two rate limit configuration groups:

• connections_rate_limit for connection based rate limiting on OSI layer 4/TCP
• requests_rate_limit for request based rate limiting on OSI layer 7/HTTP

Both groups contain roughly the same attributes:

• requests (for requests_rate_limit) and connections (for connections_rate_limit): the number of requests/connections allowed within a time window (see window_size) before further incoming requests/connections are denied/blocked
• window_size: Window size for counting connections
• table_size: Size of the stick table in which the IPs and counters are stored.
• block: Whether or not to block connections. If block is disabled (or not provided), incoming requests/connections will still be tracked in the respective stick-tables, but will not be denied.

Effects of Rate Limiting

Once a rate limit is reached, haproxy-boshrelease will no longer proxy incoming requests from the rate-limited client IP to a backend. Depending on the type of rate limiting, HAProxy will respond with one of the following:

Request based Rate Limiting

HAProxy responds to the client with HTTP Status Code: 429: Too Many Requests.

Connection based Rate Limiting

The TCP connection will be rejected. This would for example show up as Empty reply from server for a curl-client. This will not result in a log statement on the HAProxy side, which can make tracing issues more difficult.

Note: If both rate-limits are reached simultaneously (e.g. if they are configured identically and every incoming HTTP request uses a new TCP connection), connection based rate-limiting will come into effect first, resulting in a dropped TCP connection.

Configuration Examples

Note: The following examples assume only an http-in frontend is configured; an https-in frontend would behave identically.

Count Incoming Requests Only (No blocking)

Configuration (deployments/haproxy/config.yml)

config:
    # [...]
    requests_rate_limit:
        window_size: 10s
        table_size: 1m

Resulting haproxy.config

backend st_http_req_rate
    stick-table type ipv6 size 1m expire 10s store http_req_rate(10s)
# [...]
frontend http-in
    http-request track-sc1 src table st_http_req_rate

Request based Rate Limiting Fully Active

Configuration

config:
    # [...]
    requests_rate_limit:
        requests: 10
        window_size: 10s
        table_size: 1m
        block: true

Resulting haproxy.config

backend st_http_req_rate
    stick-table type ipv6 size 1m expire 10s store http_req_rate(10s)
# [...]
frontend http-in
    http-request track-sc1 src table st_http_req_rate
    http-request deny status 429 if { sc_http_req_rate(1) gt 10 }

Both Types Active

Configuration

config:
    # [...]
    requests_rate_limit:
        requests: 10
        window_size: 10s
        table_size: 1m
        block: true
    connections_rate_limit:
        connections: 10
        window_size: 10s
        table_size: 1m
        block: true

Resulting haproxy.config

backend st_http_req_rate
    stick-table type ipv6 size 1m expire 10s store http_req_rate(10s)

backend st_tcp_conn_rate
    stick-table type ipv6 size 1m expire 10s store conn_rate(10s)
# [...]
frontend http-in
    # [...]
    http-request track-sc1 src table st_http_req_rate
    http-request deny status 429 if { sc_http_req_rate(1) gt 10 }

    tcp-request content track-sc0 src table st_tcp_conn_rate
    tcp-request connection reject if { sc_conn_rate(0) gt 10}

Querying Current Stick-Table Status

To get more insight into what is going on inside HAProxy regarding its rate limits, you can query the stats socket at /var/vcap/sys/run/haproxy/stats.sock to get the raw table data:

$ echo "show table st_http_req_rate" | socat /var/vcap/sys/run/haproxy/stats.sock -
# table: st_http_req_rate, type: ip, size:10485760, used:1
0x...: key=:ffff:172.18.0.1 use=0 exp=7618 http_req_rate(10000)=10

echo "show table st_tcp_conn_rate" | socat stdio /var/vcap/sys/run/haproxy/stats.sock
# => # table: st_tcp_conn_rate, type: ipv6, size:1048576, used:2
# => 0x...: key=::ffff:203.0.113.42 use=0 exp=8123 shard=0 conn_rate(10000)=5

To find the IP with the highest connection rate, use:

echo "show table st_tcp_conn_rate" | socat stdio /var/vcap/sys/run/haproxy/stats.sock | sort -t= -k2 -rn | head -1

Note: You will likely need sudo permission to run socat.

Control Connection Rate Limiting via HAProxy Runtime API

Normally, changing rate-limit settings requires updating the manifest and reloading HAProxy. Using the HAProxy Runtime API, blocking can be enabled or disabled, and the connection threshold can be tightened or loosened while HAProxy continues running and serving traffic. This is particularly useful during an active incident, when a rapid reaction is needed.

Prerequisites

• ha_proxy.master_cli_enable: true or ha_proxy.stats_enable: true must be set in the manifest to enable the HAProxy Runtime API.
• ha_proxy.connections_rate_limit.table_size and ha_proxy.connections_rate_limit.window_size must be defined in the manifest to create the stick table and enable connection tracking.
• root permissions are required to write to the socket.

How Runtime Control Works

When HAProxy starts, it reads connections_rate_limit.block and connections_rate_limit.connections from the manifest and stores them as process-level variables inside the running HAProxy process. Updating a variable instantly changes the behavior for all subsequent connections, as every new TCP connection is evaluated against these variables in real time.

These variables are updated by sending plain-text commands to the HAProxy stats socket. The socket is available as long as HAProxy is running, and any change persists until the next redeploy, at which point the manifest values are restored.

Note: the connections threshold is applied per the defined window_size, which is also used for counting connections. For example, if window_size is set to 10s and connections is set to 100, then the threshold of 100 connections applies to every 10-second window.

Inspect Current Variable Values

echo "get var proc.connections_rate_limit_connections" | sudo socat stdio /var/vcap/sys/run/haproxy/stats.sock
# => proc.connections_rate_limit_connections: type=sint value=<600>

echo "get var proc.connections_rate_limit_block" | sudo socat stdio /var/vcap/sys/run/haproxy/stats.sock
# => proc.connections_rate_limit_block: type=bool value=<1>

Enable or Disable Blocking at Runtime

# Enable blocking (equivalent to setting block: true in the manifest)
echo "experimental-mode on; set var proc.connections_rate_limit_block bool(true)" | sudo socat stdio /var/vcap/sys/run/haproxy/stats.sock

# Disable blocking without reloading (equivalent to setting block: false in the manifest)
echo "experimental-mode on; set var proc.connections_rate_limit_block bool(false)" | sudo socat stdio /var/vcap/sys/run/haproxy/stats.sock

Adjust the Connections Threshold at Runtime

# Allow up to 100 connections per window (equivalent to setting connections: 100 in the manifest)
echo "experimental-mode on; set var proc.connections_rate_limit_connections int(100)" | sudo socat stdio /var/vcap/sys/run/haproxy/stats.sock

Enable Rate Limiting and Set Threshold in One Step

echo "experimental-mode on; set var proc.connections_rate_limit_connections int(100); set var proc.connections_rate_limit_block bool(true)" | sudo socat stdio /var/vcap/sys/run/haproxy/stats.sock