temporal-configmap-dynconfig

OSS Temporal Server ships with a file-based dynamic config client. It works, but it has real operational limits: you edit a YAML file, wait up to 10 seconds for the poll interval, and repeat that edit on every server host. In a Kubernetes deployment this is doubly painful — there is no single file to edit, changes don't propagate to running pods without a ConfigMap update, and drift between pods is invisible until something breaks.

This library replaces that client with one backed by a Kubernetes ConfigMap. All Temporal server pods watch the same ConfigMap and receive config changes simultaneously via the Kubernetes watch API — no polling, no per-pod file management, no drift. A single kubectl edit configmap (or a call to WriteConfig) propagates to every pod in the cluster within seconds.

It implements both dynamicconfig.Client and dynamicconfig.NotifyingClient, so Temporal uses push-based updates rather than polling. It is a drop-in replacement: wire it in at server startup, point it at a ConfigMap, and the rest of your server code is unchanged.

When to use this:

• You run Temporal on Kubernetes and want config changes applied simultaneously across all pods
• You want a single ConfigMap as the source of truth for dynamic config — editable with kubectl, a GitOps pipeline, or programmatically
• You want push-based config updates without standing up etcd
• You want an audit log of every config change (Kubernetes tracks ConfigMap revision history)

Table of contents

• How it works
• Prerequisites
• Repository structure
• Installation
• Configuration
• Usage
• Storing dynamic config values in the ConfigMap
• Startup behaviour
• Shutdown
• Watch resilience
• Metrics
• Differences from the OSS file-based client
• Kubernetes RBAC
• Local development

How it works

• On startup, loads the full config from the ConfigMap's data key into an in-memory map
• Creates the ConfigMap if it does not already exist, and seeds missing default values from the embedded defaults.yaml
• Opens a Kubernetes Watch stream on the ConfigMap — changes propagate immediately without polling
• Implements both dynamicconfig.Client and dynamicconfig.NotifyingClient, so Temporal uses push-based updates instead of polling
• The watch supervisor reconnects automatically on stream errors, and performs a full resync every 5 minutes to guard against missed events
• Every key change is logged at INFO with old and new values

Prerequisites

• Go 1.22+
• A Kubernetes cluster (1.21+) — or k8s.io/client-go/kubernetes/fake for unit tests
• OSS Temporal server

Repository structure

atomic.go       atomicValue[T] — typesafe sync/atomic.Value wrapper
client.go       Dynamic config client: GetValue, Subscribe, WriteConfig, DumpAll, LogAll, watch loop
config.go       Config struct, YAML tags, EnsureDefaults helper
defaults.yaml   Embedded default values seeded into the ConfigMap on first start
client_test.go  Unit tests using kubernetes/fake — no real cluster needed

Installation

This library does not import go.temporal.io/server from the public module proxy — it requires a local checkout of the Temporal server source. This is intentional: the library compiles against the same server version you are running, so there is no version skew between the dynamic config client and the server internals it integrates with.

Step 1 — check out the Temporal server at the release tag matching the version you are deploying:

git clone https://github.com/temporalio/temporal.git /path/to/temporal
cd /path/to/temporal
git checkout v1.31.0   # use the tag matching your deployment

Step 2 — in your go.mod, add replace directives for both the Temporal server and this library. Neither is published to the module proxy, so both require a local path:

replace (
    go.temporal.io/server                               => /path/to/temporal
    github.com/temporalio/temporal-configmap-dynconfig  => /path/to/temporal-configmap-dynconfig
)

Important: always point the go.temporal.io/server replace directive at a release tag checkout, not master. The master branch uses pre-release versions of go.temporal.io/api that are not published to the module proxy, which will break go mod tidy for anyone who does not also have those pre-release modules locally.

Configuration

Config is a plain Go struct — populate it directly or unmarshal it from YAML.

Minimal

cfg := configmapdynconfig.Config{
    Namespace:     "temporal",
    ConfigMapName: "temporal-dynconfig",
    DataKey:       "config.yaml",
}
cfg.EnsureDefaults()

Equivalent YAML:

namespace: temporal
configMapName: temporal-dynconfig
dataKey: config.yaml

Config fields

Field	Required	Default	Description
namespace	yes	—	Kubernetes namespace where the ConfigMap lives.
configMapName	no	temporal-dynconfig	Name of the ConfigMap.
dataKey	no	config.yaml	Key within ConfigMap.data that holds the YAML config.
resyncPeriod	no	5m	Full reload interval even without watch events. Guards against missed events.

Environment variable wiring

A typical container entrypoint sets:

Env var	Maps to	Example
TEMPORAL_K8S_NAMESPACE	namespace	temporal
DYNCONFIG_CONFIGMAP_NAME	configMapName	temporal-dynconfig
DYNCONFIG_DATA_KEY	dataKey	config.yaml

Example wiring in Go:

cfg := configmapdynconfig.Config{
    Namespace:     getenv("TEMPORAL_K8S_NAMESPACE", "temporal"),
    ConfigMapName: getenv("DYNCONFIG_CONFIGMAP_NAME", "temporal-dynconfig"),
    DataKey:       getenv("DYNCONFIG_DATA_KEY", "config.yaml"),
}
cfg.EnsureDefaults()

Usage

Wire into OSS Temporal server

The key constraint is that the ConfigMap dynconfig client and the Temporal server must share a single metrics.Handler. If you pass a separate handler to each, the server starts its own Prometheus HTTP listener that conflicts with the one already bound by the handler you gave the client — server metrics will fail to start or emit nothing.

Build the handler once from the server config, pass it to NewClient, and pass the same instance to temporal.WithCustomMetricsHandler.

package main

import (
    "context"
    "log"

    configmapdynconfig "github.com/temporalio/temporal-configmap-dynconfig"
    "go.temporal.io/server/common/config"
    temporallog "go.temporal.io/server/common/log"
    "go.temporal.io/server/common/metrics"
    "go.temporal.io/server/temporal"
    "k8s.io/client-go/kubernetes"
    "k8s.io/client-go/rest"
)

func main() {
    ctx := context.Background()

    // Load the Temporal server config.
    cfg, err := config.Load(config.WithEmbedded())
    if err != nil {
        log.Fatalf("load config: %v", err)
    }

    logger := temporallog.NewZapLogger(temporallog.BuildZapLogger(cfg.Log))

    // Build ONE shared metrics handler from the server's own metrics config.
    // Pass it to both NewClient and WithCustomMetricsHandler so they share a
    // single Prometheus registry and HTTP listener.
    metricsHandler, err := metrics.MetricsHandlerFromConfig(logger, cfg.Global.Metrics)
    if err != nil {
        log.Fatalf("create metrics handler: %v", err)
    }

    // Build in-cluster Kubernetes client.
    k8sCfg, err := rest.InClusterConfig()
    if err != nil {
        log.Fatalf("in-cluster config: %v", err)
    }
    k8sClient, err := kubernetes.NewForConfig(k8sCfg)
    if err != nil {
        log.Fatalf("k8s client: %v", err)
    }

    dynCfg := configmapdynconfig.Config{
        Namespace:     getenv("TEMPORAL_K8S_NAMESPACE", "temporal"),
        ConfigMapName: getenv("DYNCONFIG_CONFIGMAP_NAME", "temporal-dynconfig"),
        DataKey:       getenv("DYNCONFIG_DATA_KEY", "config.yaml"),
    }
    dynCfg.EnsureDefaults()

    // Tag dynconfig metrics with the service(s) this process is running.
    dcMetrics := metricsHandler.WithTags(metrics.StringTag("service_name", "frontend,history,matching,worker"))

    dcClient, err := configmapdynconfig.NewClient(
        ctx,
        k8sClient,
        dynCfg.Namespace, dynCfg.ConfigMapName, dynCfg.DataKey,
        logger,
        dcMetrics,
    )
    if err != nil {
        log.Fatalf("create configmap dynconfig client: %v", err)
    }
    defer dcClient.Stop()

    server, err := temporal.NewServer(
        temporal.WithConfig(cfg),
        temporal.WithLogger(logger),
        temporal.WithDynamicConfigClient(dcClient),
        temporal.WithCustomMetricsHandler(metricsHandler), // same handler — prevents duplicate listener
        temporal.InterruptOn(temporal.InterruptCh()),
    )
    if err != nil {
        log.Fatalf("create server: %v", err)
    }
    if err := server.Start(); err != nil {
        log.Fatalf("start server: %v", err)
    }
}

Storing dynamic config values in the ConfigMap

All dynamic config is stored in a single YAML document under the configured data key (default config.yaml). The format is the same as the OSS file-based client: each key maps to a list of constrained values.

Simple global value

# kubectl edit configmap temporal-dynconfig -n temporal
# data:
#   config.yaml: |
frontend.namespaceRPS.visibility:
  - value: 100
    constraints: {}
matching.numTaskqueueReadPartitions:
  - value: 4
    constraints: {}

Per-namespace override with global fallback

frontend.globalNamespaceRPS:
  - value: 500
    constraints:
      namespace: high-traffic-namespace
  - value: 1200
    constraints: {}

Applying changes

Edit the ConfigMap directly:

kubectl edit configmap temporal-dynconfig -n temporal

Or apply a YAML file:

kubectl apply -f dynconfig.yaml

Changes are picked up by all server pods within seconds — no restart required.

Supported constraint fields

Constraint key	Type	Description
namespace	string	Namespace name
namespaceId	string	Namespace ID
taskQueueName	string	Task queue name
taskType	string or int	Workflow or Activity
historyTaskType	string or int	Internal history task type
shardId	int	History shard ID
destination	string	Nexus destination

Temporal evaluates constraints in precedence order (most specific wins). A value with constraints: {} acts as the global default.

Writing values programmatically

import "go.temporal.io/server/common/dynamicconfig"

err := dcClient.WriteConfig(ctx,
    dynamicconfig.FrontendRPS,
    []dynamicconfig.ConstrainedValue{
        {
            Value:       500,
            Constraints: dynamicconfig.Constraints{Namespace: "high-traffic-namespace"},
        },
        {
            Value: 1200,
        },
    },
)

WriteConfig serializes the values to YAML, patches the ConfigMap, and immediately reloads the in-memory cache. Intended for CLI tooling and bootstrappers, not hot paths.

Inspecting the loaded config (DumpAll / LogAll)

OSS Temporal has no built-in way to see what dynamic config values are currently active. The ConfigMap client adds two methods for this.

DumpAll() returns a snapshot of the full in-memory map as map[string][]dynamicconfig.ConstrainedValue. The map is a copy — safe to iterate after the client is stopped:

snapshot := dcClient.DumpAll()
for key, values := range snapshot {
    fmt.Printf("%s: %+v\n", key, values)
}

Typical uses:

• Expose it from a debug HTTP handler so you can curl the live state
• Log it at startup to confirm all expected overrides were loaded
• Diff two snapshots to see what changed between deployments

LogAll() writes every key and its constrained values to the logger at INFO level — one log line per key. Useful as a startup diagnostic without any extra wiring:

// call once after NewClient returns, before starting the server
dcClient.LogAll()

Example output (structured logging):

dynamic config dump start   totalKeys=12
dynamic config entry        key=frontend.namespacerps.visibility  values=[{constraints:{} value:100}]
dynamic config entry        key=matching.numtaskqueuereadpartitions  values=[{constraints:{} value:4}]
...
dynamic config dump end

Both methods read directly from the same atomic in-memory map that GetValue uses — no Kubernetes API round-trip, no lock contention.

Deleting a value (reverts to compiled-in default)

Remove the key from config.yaml in the ConfigMap. All server pods will revert to the compiled-in default value within seconds.

Listing all current dynamic config values

kubectl get configmap temporal-dynconfig -n temporal -o jsonpath='{.data.config\.yaml}'

Startup behaviour

NewClient performs the following on startup, in order:

1. Creates the ConfigMap if it does not exist (with an empty config.yaml key)
2. Loads all values from the ConfigMap into the in-memory cache
3. Seeds any keys from the embedded defaults.yaml that are not already present in the ConfigMap — existing keys are never touched
4. Starts the background watch loop

If the ConfigMap cannot be read (e.g. RBAC is not set up correctly), NewClient returns an error — the server should not start with a broken config backend.

Shutdown

defer dcClient.Stop()  // cancels the watch goroutine and background loops

Watch resilience

The watch supervisor handles:

Event	Behaviour
Watch channel closed	Reload all values, reopen Watch
Watch error event	Reload all values, reopen Watch
Context cancellation (Stop())	Exits cleanly, no reload
No events for resyncPeriod (default 5m)	Full reload as a safety net

Backoff on reload failure: 100ms → doubles each attempt → caps at 30s.

Metrics

The client emits metrics through the same metrics.Handler the Temporal server already uses — Prometheus, OpenTelemetry, or any other backend your server is configured with.

You must share a single handler between the ConfigMap client and the Temporal server. Build it once with metrics.MetricsHandlerFromConfig, pass it to NewClient, and pass the same instance to temporal.WithCustomMetricsHandler. Without WithCustomMetricsHandler, the server starts its own Prometheus HTTP listener that conflicts with the one already bound by the handler you passed to NewClient — server metrics will fail to start or emit nothing.

metricsHandler, err := metrics.MetricsHandlerFromConfig(logger, cfg.Global.Metrics)

dcClient, err := configmapdynconfig.NewClient(ctx, k8sClient,
    namespace, configMapName, dataKey,
    logger,
    metricsHandler.WithTags(metrics.StringTag("service_name", "frontend")),
)

server, err := temporal.NewServer(
    temporal.WithDynamicConfigClient(dcClient),
    temporal.WithCustomMetricsHandler(metricsHandler), // same handler — no duplicate listener
    // ...
)

Pass metrics.NoopMetricsHandler to NewClient to disable dynconfig metrics entirely.

Emitted metrics

All metrics inherit any tags set on the handler passed to NewClient.

Metric	Type	Tags	Description
dynconfig_key_updates_total	counter	operation (DynamicConfigUpdate), key (config key name)	Incremented on every key change received from the ConfigMap watch.
dynconfig_watch_reconnects_total	counter	reason (stream_ended)	Incremented whenever the watch supervisor reconnects. A spike here indicates Kubernetes API server instability.
dynconfig_watch_active	gauge	—	1 while the watch stream is running, 0 while stopped or reconnecting. Alert on this going to 0.
dynconfig_keys_loaded	gauge	—	Number of keys in the in-memory map after each full reload.
dynconfig_load_duration_seconds	timer	—	Time taken for a full ConfigMap read, on startup and each reconnect.
dynconfig_write_total	counter	result (success, error)	Outcome of each WriteConfig call.

Useful alert queries

# Watch is down on any pod — config changes are not propagating
dynconfig_watch_active{service=~"frontend|history|matching|worker"} == 0

# Frequent watch reconnects — Kubernetes API server is unstable
rate(dynconfig_watch_reconnects_total[5m]) > 0.1

Differences from the OSS file-based client

	File-based client	ConfigMap client
Update latency	Poll interval (default 10s)	Near-realtime via Kubernetes watch
Write path	Edit file on disk (per host)	kubectl edit or WriteConfig() — single source of truth
Multi-pod consistency	Depends on shared volume or config management	All pods in the cluster see the same value simultaneously
Resilience	File must be present at startup	Fails fast if ConfigMap cannot be read at startup; survives watch disruptions at runtime
Audit log	None	Kubernetes tracks ConfigMap revision history
Infrastructure dependency	Filesystem / config management tooling	Kubernetes API server (already present)

Kubernetes RBAC

Server pods need read and watch access to the ConfigMap. WriteConfig additionally requires update and patch. Apply a Role and RoleBinding:

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: temporal-dynconfig-reader
  namespace: temporal
rules:
  - apiGroups: [""]
    resources: ["configmaps"]
    resourceNames: ["temporal-dynconfig"]
    verbs: ["get", "watch", "list", "update", "patch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: temporal-dynconfig-reader
  namespace: temporal
subjects:
  - kind: ServiceAccount
    name: temporal-server
    namespace: temporal
roleRef:
  kind: Role
  name: temporal-dynconfig-reader
  apiGroup: rbac.authorization.k8s.io

If you only want read-only access (no WriteConfig), remove update and patch from the verbs list.

Local development

For unit tests, use the kubernetes/fake client — no real cluster or running Kubernetes API server needed:

import (
    "k8s.io/client-go/kubernetes/fake"
    configmapdynconfig "github.com/temporalio/temporal-configmap-dynconfig"
)

fakeClient := fake.NewSimpleClientset()
client, err := configmapdynconfig.NewClient(
    context.Background(),
    fakeClient,
    "temporal", "temporal-dynconfig", "config.yaml",
    log.NewNoopLogger(),
    metrics.NoopMetricsHandler,
)

All watch events, ConfigMap creates, and updates work correctly against the fake client. See client_test.go for examples.