CONFIGURATION.md

July 3, 2025 ยท View on GitHub

Client global configs

C/P legend: C = Consumer, P = Producer, * = both

PropertyC/PRangeDefaultDescription
builtin_features*gzip, snappy, ssl, sasl, regex, lz4, sasl_gssapi, sasl_plain, sasl_scram, plugins, zstd, sasl_oauthbearerIndicates the builtin features for this build of librdkafka. An application can either query this value or attempt to set it with its list of required features to check for library support.
client_id*rdkafkaClient identifier
bootstrap_servers*Initial list of brokers host:port separated by comma
connections_max_idle_ms*0 .. 21474836470Close broker connections after the specified time of inactivity. Disable with 0. If this property is left at its default value some heuristics are performed to determine a suitable default value, this is currently limited to identifying brokers on Azure (see librdkafka issue #3109 for more info).
message_max_bytes*1000 .. 10000000001000000Maximum Kafka protocol request message size. Due to differing framing overhead between protocol versions the producer is unable to reliably enforce a strict max message limit at produce time and may exceed the maximum size by one message in protocol ProduceRequests, the broker will enforce the the topic's max_message_bytes limit (see Apache Kafka documentation).
message_copy_max_bytes*0 .. 100000000065535Maximum size for message to be copied to buffer. Messages larger than this will be passed by reference (zero-copy) at the expense of larger iovecs.
receive_message_max_bytes*1000 .. 2147483647100000000Maximum Kafka protocol response message size. This serves as a safety precaution to avoid memory exhaustion in case of protocol hickups. This value must be at least fetch_max_bytes + 512 to allow for protocol overhead; the value is adjusted automatically unless the configuration property is explicitly set.
max_in_flight*1 .. 10000001000000Maximum number of in-flight requests per broker connection. This is a generic property applied to all broker communication, however it is primarily relevant to produce requests. In particular, note that other mechanisms limit the number of outstanding consumer fetch request per broker to one.
metadata_request_timeout_ms*10 .. 90000060000Non-topic request timeout in milliseconds. This is for metadata requests
topic_metadata_refresh_interval_ms*-1 .. 3600000300000Topic metadata refresh interval in milliseconds. The metadata is automatically refreshed on error and connect. Use -1 to disable the intervalled refresh.
metadata_max_age_ms*1 .. 86400000900000Metadata cache max age. Defaults to topic_metadata_refresh_interval_ms * 3
topic_metadata_refresh_fast_interval_ms*1 .. 60000250When a topic loses its leader a new metadata request will be enqueued with this initial interval, exponentially increasing until the topic metadata has been refreshed. This is used to recover quickly from transitioning leader brokers
topic_metadata_refresh_sparse*true, falsetrueSparse metadata requests (consumes less network bandwidth)
topic_metadata_propagation_max_ms*0 .. 360000030000Apache Kafka topic creation is asynchronous and it takes some time for a new topic to propagate throughout the cluster to all brokers. If a client requests topic metadata after manual topic creation but before the topic has been fully propagated to the broker the client is requesting metadata from, the topic will seem to be non-existent and the client will mark the topic as such, failing queued produced messages with ERR__UNKNOWN_TOPIC. This setting delays marking a topic as non-existent until the configured propagation max time has passed. The maximum propagation time is calculated from the time the topic is first referenced in the client.
topic_blacklist*Topic blacklist, a comma-separated list of regular expressions for matching topic names that should be ignored in broker metadata information as if the topics did not exist
debug*generic, broker, topic, metadata, feature, queue, msg, protocol, cgrp, security, fetch, interceptor, plugin, consumer, admin, eos, mock, assignor, conf, all
socket_timeout_ms*10 .. 30000060000Timeout for network requests
socket_send_buffer_bytes*0 .. 1000000000Broker socket send buffer size. System default is used if 0
socket_receive_buffer_bytes*0 .. 1000000000Broker socket receive buffer size. System default is used if 0
socket_keepalive_enable*true, falsefalseEnable TCP keep-alives (SO_KEEPALIVE) on broker sockets
socket_nagle_disable*true, falsefalseDisable the Nagle algorithm on broker sockets (TCP_NODELAY)
socket_max_fails*0 .. 10000001Disconnect from broker when this number of send failures (e.g., timed out requests) is reached. Disable with 0. NOTE: The connection is automatically re-established
broker_address_ttl*0 .. 864000001000How long to cache the broker address resolving results (milliseconds)
broker_address_family*any, v4, v6anyAllowed broker IP address families: any, v4, v6
reconnect_backoff_ms*0 .. 3600000100The initial time to wait before reconnecting to a broker after the connection has been closed. The time is increased exponentially until reconnect.backoff.max.ms is reached. -25% to +50% jitter is applied to each reconnect backoff. A value of 0 disables the backoff and reconnects immediately.
reconnect_backoff_max_ms*0 .. 360000010000The maximum time to wait before reconnecting to a broker after the connection has been closed.
statistics_interval_ms*0 .. 864000000Statistics emit interval. The application also needs to register a stats callback using stats_callback config. The granularity is 1000ms. A value of 0 disables statistics.
enabled_events*0 .. 21474836470Enable event sourcing. A bitmask of events to enable for consuming.
stats_callback*module or fun/2undefinedA callback where stats are sent
log_level*0 .. 76Logging level (syslog(3) levels)
log_queue*true, falsefalseDisable spontaneous log_cb from internal librdkafka threads, instead enqueue log messages on queue set with rd_kafka_set_log_queue() and serve log callbacks or events through the standard poll APIs. NOTE: Log messages will linger in a temporary queue until the log queue has been set.
log_thread_name*true, falsetruePrint internal thread name in log messages (useful for debugging librdkafka internals).
enable_random_seed*true, falsetrueIf enabled librdkafka will initialize the PRNG with srand(current_time.milliseconds) on the first invocation of rd_kafka_new() (required only if rand_r() is not available on your platform). If disabled the application must call srand() prior to calling rd_kafka_new().
log_connection_close*true, falsetrueLog broker disconnects. It might be useful to turn this off when interacting with 0.9 brokers with an aggressive connection_max_idle_ms value.
api_version_request*true, falsetrueRequest broker's supported API versions to adjust functionality to available protocol features. If set to false, or the ApiVersionRequest fails, the fallback version broker_version_fallback will be used. NOTE: Depends on broker version >=0.10.0. If the request is not supported by (an older) broker the broker_version_fallback fallback is used
api_version_request_timeout_ms*1 .. 30000010000Timeout for broker API version requests.
api_version_fallback_ms*0 .. 6048000000Dictates how long the broker_version_fallback fallback is used in the case the ApiVersionRequest fails. NOTE: The ApiVersionRequest is only issued when a new connection to the broker is made (such as after an upgrade)
broker_version_fallback*0.10.0Older broker versions (<0.10.0) provides no way for a client to query for supported protocol features (ApiVersionRequest, see api_version_request) making it impossible for the client to know what features it may use. As a workaround a user may set this property to the expected broker version and the client will automatically adjust its feature set accordingly if the ApiVersionRequest fails (or is disabled). The fallback broker version will be used for api_version_fallback_ms. Valid values are: 0.9.0, 0.8.2, 0.8.1, 0.8.0.
security_protocol*plaintext, ssl, sasl_plaintext, sasl_sslplaintextProtocol used to communicate with brokers
ssl_cipher_suites*A cipher suite is a named combination of authentication, encryption, MAC and key exchange algorithm used to negotiate the security settings for a network connection using TLS or SSL network protocol. See manual page for ciphers(1) and `SSL_CTX_set_cipher_list(3)
ssl_curves_list*The supported-curves extension in the TLS ClientHello message specifies the curves (standard/named, or 'explicit' GF(2^k) or GF(p)) the client is willing to have the server use. See manual page for SSL_CTX_set1_curves_list(3). OpenSSL >= 1.0.2 required.
ssl_sigalgs_list*The client uses the TLS ClientHello signature_algorithms extension to indicate to the server which signature/hash algorithm pairs may be used in digital signatures. See manual page for SSL_CTX_set1_sigalgs_list(3). OpenSSL >= 1.0.2 required.
ssl_key_location*Path to client's private key (PEM) used for authentication.
ssl_key_password*Private key passphrase
ssl_key_pem*Client's private key string (PEM format) used for authentication.
ssl_certificate_location*Path to client's public key (PEM) used for authentication.
ssl_certificate_pem*Client's public key string (PEM format) used for authentication.
ssl_ca_location*File or directory path to CA certificate(s) for verifying the broker's key.
ssl_crl_location*Path to CRL for verifying broker's certificate validity.
ssl_keystore_location*Path to client's keystore (PKCS#12) used for authentication.
ssl_keystore_password*Client's keystore (PKCS#12) password.
enable_ssl_certificate_verification*true, falsetrueEnable OpenSSL's builtin broker (server) certificate verification.
ssl_endpoint_identification_algorithm*none, httpshttpsEndpoint identification algorithm to validate broker hostname using broker certificate. https - Server (broker) hostname verification as specified in RFC2818. none - No endpoint verification. OpenSSL >= 1.0.2 required.
sasl_mechanisms*GSSAPISASL mechanism to use for authentication. Supported: GSSAPI, PLAIN, SCRAM-SHA-256, SCRAM-SHA-512. NOTE: Despite the name only one mechanism must be configured.
sasl_kerberos_service_name*kafkaKerberos principal name that Kafka runs as
sasl_kerberos_principal*kafkaclientThis client's Kerberos principal name.
sasl_kerberos_kinit_cmd*Full kerberos kinit command string, %{config.prop.name} is replaced by corresponding config object value, %{broker.name} returns the broker's hostname
sasl_kerberos_keytab*Path to Kerberos keytab file. Uses system default if not set.NOTE: This is not automatically used but must be added to the template in sasl_kerberos_kinit_cmd as ... -t %{sasl_kerberos_keytab}
sasl_kerberos_min_time_before_relogin*0 .. 8640000060000Minimum time in milliseconds between key refresh attempts.
sasl_username*SASL username for use with the PLAIN and SASL-SCRAM-.. mechanisms
sasl_password*SASL password for use with the PLAIN and SASL-SCRAM-.. mechanism
sasl_oauthbearer_config*SASL/OAUTHBEARER configuration. The format is implementation-dependent and must be parsed accordingly. The default unsecured token implementation (see https://tools.ietf.org/html/rfc7515#appendix-A.5) recognizes space-separated name=value pairs with valid names including principalClaimName, principal, scopeClaimName, scope, and lifeSeconds. The default value for principalClaimName is "sub", the default value for scopeClaimName is "scope", and the default value for lifeSeconds is 3600. The scope value is CSV format with the default value being no/empty scope. For example: principalClaimName=azp principal=admin scopeClaimName=roles scope=role1,role2 lifeSeconds=600. In addition, SASL extensions can be communicated to the broker via extension_NAME=value. For example: principal=admin extension_traceId=123.
enable_sasl_oauthbearer_unsecure_jwt*true, falsefalseEnable the builtin unsecure JWT OAUTHBEARER token handler if no oauthbearer_refresh_cb has been set. This builtin handler should only be used for development or testing, and not in production.
oauthbearer_token_refresh_callback*module or fun/2undefinedA callback to implement SASL/OAUTHBEARER token refresh.
sasl_oauthbearer_method*default, oidcdefaultSet to "default" or "oidc" to control which login method to be used. If set to "oidc", the following properties must also be be specified: sasl_oauthbearer_client_id, sasl_oauthbearer_client_secret, and sasl_oauthbearer_token_endpoint_url.
sasl_oauthbearer_client_id*Public identifier for the application. Must be unique across all clients that the authorization server handles. Only used when sasl_oauthbearer_method is set to "oidc".
sasl_oauthbearer_client_secret*Client secret only known to the application and the authorization server. This should be a sufficiently random string that is not guessable. Only used when sasl_oauthbearer_method is set to "oidc".
sasl_oauthbearer_scope*Client use this to specify the scope of the access request to the broker. Only used when sasl_oauthbearer_method is set to "oidc".
sasl.oauthbearer.extensions*Allow additional information to be provided to the broker. Comma-separated list of key=value pairs. E.g., "supportFeatureX=true,organizationId=sales-emea".Only used when sasl_oauthbearer_method is set to "oidc".
sasl_oauthbearer_token_endpoint_url*OAuth/OIDC issuer token endpoint HTTP(S) URI used to retrieve token. Only used when sasl_oauthbearer_method is set to "oidc".
plugin_library_paths*undefinedPath where librdkafka plugins are located
group_instance_idCEnable static group membership. Static group members are able to leave and rejoin a group within the configured session.timeout.ms without prompting a group rebalance. This should be used in combination with a larger session.timeout.ms to avoid group rebalances caused by transient unavailability (e.g. process restarts). Requires broker version >= 2.3.0.
partition_assignment_strategyCrange, roundrobinName of partition assignment strategy to use when elected group leader assigns partitions to group members
session_timeout_msC1 .. 360000010000Client group session and failure detection timeout
heartbeat_interval_msC1 .. 36000003000Group session keepalive heartbeat interval
group_protocol_typeCconsumerGroup protocol type. NOTE: Currently, the only supported group protocol type is consumer.
coordinator_query_interval_msC1 .. 3600000600000How often to query for the current client group coordinator. If the currently assigned coordinator is down the configured query interval will be divided by ten to more quickly recover in case of coordinator reassignment
max_poll_interval_msC1 .. 86400000300000Maximum allowed time between calls to consume messages (e.g., rd_kafka_consumer_poll()) for high-level consumers. If this interval is exceeded the consumer is considered failed and the group will rebalance in order to reassign the partitions to another consumer group member. Warning: Offset commits may be not possible at this point. The interval is checked two times per second. See KIP-62 for more information.
auto_commit_interval_msC0 .. 864000005000The frequency in milliseconds that the consumer offsets are committed (written) to offset storage. (0 = disable). This setting is used by the high-level consumer
queued_min_messagesC1 .. 10000000100000Minimum number of messages per topic+partition in the local consumer queue
queued_max_messages_kbytesC1 .. 100000000065536Maximum number of kilobytes per topic+partition in the local consumer queue. This value may be overshot by fetch_message_max_bytes
fetch_wait_max_msC0 .. 300000500Maximum time the broker may wait to fill the response with fetch_min_bytes
fetch_queue_backoff_msC0 .. 3000001000How long to postpone the next fetch request for a topic+partition in case the current fetch queue thresholds (queued_min_messages or queued_max_messages_kbytes) have been exceded. This property may need to be decreased if the queue thresholds are set low and the application is experiencing long (~1s) delays between messages. Low values may increase CPU utilization.
fetch_message_max_bytesC1 .. 10000000001048576Initial maximum number of bytes per topic+partition to request when fetching messages from the broker. If the client encounters a message larger than this value it will gradually try to increase it until the entire message can be fetched
fetch_max_bytesC0 .. 214748313552428800Maximum amount of data the broker shall return for a Fetch request. Messages are fetched in batches by the consumer and if the first message batch in the first non-empty partition of the Fetch request is larger than this value, then the message batch will still be returned to ensure the consumer can make progress. The maximum message batch size accepted by the broker is defined via message_max_bytes (broker config) or max_message_bytes (broker topic config). fetch_max_bytes is automatically adjusted upwards to be at least message_max_bytes (consumer config).
fetch_min_bytesC1 .. 1000000001Minimum number of bytes the broker responds with. If fetch.wait.max.ms expires the accumulated data will be sent to the client regardless of this setting
fetch_error_backoff_msC0 .. 300000500How long to postpone the next fetch request for a topic+partition in case of a fetch error
isolation_levelCread_uncommitted, read_committedread_committedControls how to read messages written transactionally: read_committed - only return transactional messages which have been committed. read_uncommitted - return all messages, even transactional messages which have been aborted.
check_crcsCtrue, falsefalseVerify CRC32 of consumed messages, ensuring no on-the-wire or on-disk corruption to the messages occurred. This check comes at slightly increased CPU usage
allow_auto_create_topics*true, falsefalseAllow automatic topic creation on the broker when subscribing to or assigning non-existent topics. The broker must also be configured with auto.create.topics.enable=true for this configuration to take effect. Note: the default value (true) for the producer is different from the default value (false) for the consumer. Requires broker version >= 0.11.0.0, for older broker versions only the broker configuration applies.
client_rack*A rack identifier for this client. This can be any string value which indicates where this client is physically located. It corresponds with the broker config broker.rack.
transactional_idPEnables the transactional producer. The transactional.id is used to identify the same transactional producer instance across process restarts. It allows the producer to guarantee that transactions corresponding to earlier instances of the same producer have been finalized prior to starting any new transactions, and that any zombie instances are fenced off. If no transactional.id is provided, then the producer is limited to idempotent delivery (if enable.idempotence is set). Requires broker version >= 0.11.0.
transaction.timeout.msP1000 .. 214748364760000The maximum amount of time in milliseconds that the transaction coordinator will wait for a transaction status update from the producer before proactively aborting the ongoing transaction. If this value is larger than the transaction.max.timeout.ms setting in the broker, the init_transactions() call will fail with ERR_INVALID_TRANSACTION_TIMEOUT. The transaction timeout automatically adjusts message.timeout.ms and socket.timeout.ms, unless explicitly configured in which case they must not exceed the transaction timeout (socket.timeout.ms must be at least 100ms lower than transaction.timeout.ms). This is also the default timeout value if no timeout (-1) is supplied to the transactional API methods.
enable_idempotencePtrue, falsefalseWhen set to true, the producer will ensure that messages are successfully produced exactly once and in the original produce order. The following configuration properties are adjusted automatically (if not modified by the user) when idempotence is enabled: max_in_flight_requests_per_connection=5 (must be less than or equal to 5), retries=INT32_MAX (must be greater than 0), acks=all, queuing_strategy=fifo. Producer instantation will fail if user-supplied configuration is incompatible.
enable_gapless_guaranteePtrue, falsefalseWhen set to true, any error that could result in a gap in the produced message series when a batch of messages fails, will raise a fatal error (ERR__GAPLESS_GUARANTEE) and stop the producer. Requires enable_idempotence=true.
queue_buffering_max_messagesP0 .. 2147483647100000Maximum number of messages allowed on the producer queue. This queue is shared by all topics and partitions. A value of 0 disables this limit.
queue_buffering_max_kbytesP1 .. 21474836471048576Maximum total message size sum allowed on the producer queue. This queue is shared by all topics and partitions. This property has higher priority than queue_buffering_max_messages.
queue_buffering_max_msP0 .. 9000005Maximum time, in milliseconds, for buffering data on the producer queue
message_send_max_retriesP0 .. 21474836472147483647How many times to retry sending a failing MessageSet. Note: retrying may cause reordering
retry_backoff_msP1 .. 300000100The backoff time in milliseconds before retrying a message send
queue_buffering_backpressure_thresholdP1 .. 10000001The threshold of outstanding not yet transmitted broker requests needed to backpressure the producer's message accumulator. If the number of not yet transmitted requests equals or exceeds this number, produce request creation that would have otherwise been triggered (for example, in accordance with linger.ms) will be delayed. A lower number yields larger and more effective batches. A higher value can improve latency when using compression on slow machines.
queue_buffering_overflow_strategyPlocal_disk_queue, block_calling_process, drop_recordslocal_disk_queueWhat strategy to pick in case the memory queue is full: write messages on a local disk queue and send them in kafka when the in memory queue have enough space, block calling process until memory queue has enough space or drop the messages
compression_codecPnone, gzip, snappy, lz4, zstdnonecompression codec to use for compressing message sets. This is the default value for all topics, may be overriden by the topic configuration property compression_codec
batch_num_messagesP1 .. 100000010000Maximum number of messages batched in one MessageSet. The total MessageSet size is also limited by message_max_bytes
batch_sizeP1 .. 21474836471000000Maximum size (in bytes) of all messages batched in one MessageSet, including protocol framing overhead. This limit is applied after the first message has been added to the batch, regardless of the first message's size, this is to ensure that messages that exceed batch.size are produced. The total MessageSet size is also limited by batch_num_messages and message_max_bytes.
delivery_report_only_errorPtrue, falsefalseOnly provide delivery reports for failed messages
delivery_report_callbackPmodule or fun/2undefinedA callback where delivery reports are sent (erlkaf_producer_callbacks behaviour)
sticky_partitioning_linger_msP0 .. 90000010Delay in milliseconds to wait to assign new sticky partitions for each topic. By default, set to double the time of linger.ms. To disable sticky behavior, set to 0. This behavior affects messages with the key NULL in all cases, and messages with key lengths of zero when the consistent_random partitioner is in use. These messages would otherwise be assigned randomly. A higher value allows for more effective batching of these messages.
local_queue_pathPPath to directory where local disk queue files will be paced if queue_buffering_overflow_strategy for producer set to local_disk_queue. Default value is priv directory of erlkaf application.
client_dns_lookup*use_all_dns_ips, resolve_canonical_bootstrap_servers_onlyuse_all_dns_ipsControls how the client uses DNS lookups. By default, when the lookup returns multiple IP addresses for a hostname, they will all be attempted for connection before the connection is considered failed. This applies to both bootstrap and advertised servers. If the value is set to resolve_canonical_bootstrap_servers_only, each entry will be resolved and expanded into a list of canonical names.

Topic configuration properties

C/P legend: C = Consumer, P = Producer, * = both

PropertyC/PRangeDefaultDescription
request_required_acksP-1 .. 1000-1This field indicates how many acknowledgements the leader broker must receive from ISR brokers before responding to the request: 0=Broker does not send any response/ack to client, 1=Only the leader broker will need to ack the message, -1 or all=broker will block until message is committed by all in sync replicas (ISRs) or broker's in.sync.replicas setting before sending response.
request_timeout_msP1 .. 214748364730000The ack timeout of the producer request in milliseconds. This value is only enforced by the broker and relies on request_required_acks being != 0
message_timeout_msP0 .. 21474836470Local message timeout. This value is only enforced locally and limits the time a produced message waits for successful delivery. A time of 0 is infinite
partitionerPrandom,consistent,consistent_random, murmur2, murmur2_randomconsistent_randomPartitioner: random - random distribution, consistent - CRC32 hash of key (Empty and NULL keys are mapped to single partition), consistent_random - CRC32 hash of key (Empty and NULL keys are randomly partitioned), murmur2 - Java Producer compatible Murmur2 hash of key (NULL keys are mapped to single partition), murmur2_random - Java Producer compatible Murmur2 hash of key (NULL keys are randomly partitioned. This is functionally equivalent to the default partitioner in the Java Producer.).
compression_codecPnone, gzip, snappy, lz4, inherit, zstdinheritCompression codec to use for compressing message sets
compression_levelP-1 .. 12-1Compression level parameter for algorithm selected by configuration property compression_codec. Higher values will result in better compression at the cost of more CPU usage. Usable range is algorithm-dependent: [0-9] for gzip; [0-12] for lz4; only 0 for snappy; -1 = codec-dependent default compression level.
auto_commit_interval_msC10 .. 8640000060000The frequency in milliseconds that the consumer offsets are committed (written) to offset storage. This setting is used by the low-level legacy consumer
auto_offset_resetCsmallest, earliest, beginning, largest, latest, end, errorlargestAction to take when there is no initial offset in offset store or the desired offset is out of range: 'smallest','earliest' - automatically reset the offset to the smallest offset, 'largest','latest' - automatically reset the offset to the largest offset, 'error' - trigger an error.
offset_store_pathC.Path to local file for storing offsets. If the path is a directory a filename will be automatically generated in that directory based on the topic and partition
offset_store_sync_interval_msC-1 .. 86400000-1fsync() interval for the offset file, in milliseconds. Use -1 to disable syncing, and 0 for immediate sync after each write
consume_callback_max_messagesC0 .. 10000000Maximum number of messages to dispatch in one rd_kafka_consume_callback*() call (0 = unlimited).
dispatch_modeCone_by_one, {batch, integer()}one_by_oneThis field indicates how messages are dispatched to the message handler. One message per callback call (default) or multiple messages up to the specified size (batch). If the batch approach is used the consumer will mark as processed the last offset form the batch.
max_retriesCinfinity, non_negative_integer()infinitySpecifies how many times to retry processing a message upon failure before invoking the handle_failed_message callback, which can be used to forward the message to a Dead Letter Queue. This value also determines the number of retry attempts within the handle_failed_message callback itself, if no successful response is returned.
poll_idle_msC0 .. 864000001000This field specifies the idle time (in milliseconds) that the consumer will wait to schedule a new poll after an empty poll.