How to upgrade ClickHouse Operator
June 18, 2026 · View on GitHub
Note: Before you upgrade check releases notes if there are any backward incompatible changes between your version and the latest version.
Supported upgrade path: previous minor → current minor (e.g. 0.26 → 0.27). Skipping multiple minor versions is not validated in CI. To upgrade from a much older version, perform incremental minor-version upgrades. For installations running ClickHouse Keeper (CHK) on operator 0.23.x, see the manual PV-migration procedure in keeper_migration_from_23_to_24.md — direct 0.23 → current is no longer covered by CI.
0.27.0 → 0.27.1 FIPS posture: the default altinity/clickhouse-operator and altinity/metrics-exporter images are now FIPS 140-3 compatible (not certified). They are built with GOFIPS140=v1.0.0 against Go's native FIPS 140-3 cryptographic module (currently in CMVP review) and ship with runtime GODEBUG=fips140=off — the FIPS module is linked into the binary but dormant by default, preserving pre-0.27.1 runtime behavior on upgrade. Customers opt the module into active mode at runtime via kubectl set env deploy/clickhouse-operator -n <ns> GODEBUG=fips140=on --containers='*' (permissive: TLS filtering, non-FIPS calls allowed) or GODEBUG=fips140=only (strict: non-FIPS calls panic). Mirror the command for the metrics-exporter deployment. The build-time default is governed by the new GODEBUG_FIPS140 Docker ARG (accepted: off/on/only/debug; default off); rebuild with --build-arg GODEBUG_FIPS140=on (or =only) to bake an active-FIPS mode into a customer-specific image. The operator's identifier-derivation paths use inline pure-Go SHA-1/MD5 implementations (no crypto/sha1 / crypto/md5 imports) so they remain compatible under every mode including only. Customer prerequisite (only when activating FIPS): when GODEBUG=fips140=on or =only is set, the kubeconfig CA, ClickHouse server certificates, and ZooKeeper/Keeper certificates the operator dials must be SHA-256-or-later signed — SHA-1-signed certs will fail the very first TLS handshake. With the default fips140=off, this prerequisite does not apply. See security_hardening_fips.md for details.
Release evidence (new for 0.27.1): the build_branch.yaml CI workflow now publishes a release-evidence-<version> artifact alongside each tag-push image build, containing the image digest, syft SPDX SBOM, multi-arch manifest, and buildx metadata for both clickhouse-operator and metrics-exporter. Artifacts are also attached to the GitHub Release page. Inline SBOM and SLSA provenance attestations are embedded in the image manifest itself (via docker buildx --sbom=true --provenance=mode=max). Auditors verifying compliance with the FIPS 140-3 release gate can fetch these without rebuilding. See docs/fips_evidence_verification.md for verification recipes.
0.27.0 → 0.27.1 new chopconf knobs: the security.* section gains per-component TLS toggles for the operator's own outbound connections — security.clickhouse.tls.{verify,minVersion,serverName,rootCA,rootCASecretRef} (ClickHouse), security.zookeeper.tls.{verify,minVersion} (ZooKeeper/Keeper), and security.kubernetes.tls.{verify,minVersion} (K8s API client — verify is a load-time gate, minVersion is CRD-only and not yet wired to the transport). It also adds security.ipc.{mode,bindHost,tokenPath} to harden the operator↔metrics-exporter REST channel, plus three orthogonal hardening axes: the TLS-hardening master switch security.policy (Permissive default, Enforced coerces all transport knobs to Strict and rejects FIPS-incompatible CR postures — TLS only, no longer Fatals on non-FIPS-built binaries), the FIPS cryptographic-module gate security.fips.enforced (false default, true Fatals at startup if the binary was not built with GOFIPS140 and triggers the same TLS coercions as Enforced), and the workload supply-chain gate security.images.policy (Permissive default, FIPSRequired rejects CRs whose ClickHouse/Keeper images lack fips in their tag).
0.27.0 → 0.27.1 FIPS-compatible non-security hash sites: the two non-security hash sites used for deterministic-identifier derivation — pkg/util/hash.go::HashIntoString (the K8s clickhouse.altinity.com/object-version label fingerprint) and pkg/util/shell.go::BuildShellEnvVarName (the long-name uniqueness suffix for shell env-var names) — now compute their digests via inline pure-Go bitwise implementations rather than via crypto/sha1 / crypto/md5. The byte-level outputs are unchanged (the inline code follows FIPS PUB 180-4 §6.1.2 / RFC 3174 and RFC 1321 respectively, cited only as reference standards for byte-output compatibility, not as cryptographic protection), so existing object-version labels and env-var names stay byte-identical across the upgrade. No StatefulSet rollout is triggered by this change. Because these inline impls do not invoke any forbidden primitive, the operator binary runs cleanly under every GODEBUG=fips140 mode (the shipped default is off; customers can opt into on / only / debug at runtime via Pod env). The byte-identical guarantee means switching modes never re-hashes a K8s object's label or env-var name on upgrade. See security_hardening_fips.md § "Runtime mode" for the full mode table and § "Non-security hash exclusions" for the FIPS scope-spec authorization of non-security hashing outside the cryptographic boundary.
0.27.0 → 0.27.1 strict-FIPS coercions: when EITHER security.policy: Enforced OR security.fips.enforced: true is set, the operator additionally (a) coerces every per-component TLS knob to its Strict position at startup, (b) re-registers the legacy ClickHouse TLS config with InsecureSkipVerify=false so DSNs without explicit security knobs still get verified TLS, (c) coerces clickhouse.access.scheme: http → https and rejects plain HTTP, (d) rejects ZK digest: auth files (the vendored go-zookeeper digest hashes passwords with SHA-1), and (e) rejects CHIs/CHKs that reference plain-text external ZooKeeper (i.e. zookeeper.nodes[*].secure not set). security.fips.enforced: true additionally Fatals at startup if the binary was not built with GOFIPS140 (the cryptographic-module assertion); security.policy: Enforced alone no longer fires that gate. Review your manifests before flipping either switch.
0.27.0 → 0.27.1 FIPS startup banner enhancement: both the operator and metrics-exporter now emit an additional FIPS env: log line at startup, immediately after the existing FIPS: banner. It echoes the raw GODEBUG environment variable, the DefaultGODEBUG build setting baked into the binary, and the GOFIPS140 build setting — e.g. FIPS env: GODEBUG=fips140=on DefaultGODEBUG=fips140=on GOFIPS140=v1.0.0. This lets operators verify at a glance that the running container actually received the GODEBUG mode they configured (vs. silently falling back to the binary default) and is the recommended diagnostic to read before filing FIPS-posture questions. New e2e test test_010078 additionally verifies that submitting security.clickhouse.tls.verify: None under security.policy: Enforced is rejected at admission with [FIPSValidationFailed] — the master switch cannot be subverted by a per-CHI knob. See security_hardening.md for full banner-output semantics.
ACVP responder (optional, build-tagged): a NIST ACVP test responder can be embedded into operator and metrics-exporter binaries via the acvp_wrapper build tag. Default builds do NOT include the responder. See docs/security_hardening_fips.md § ACVP for the test-evidence pipeline.
0.27.0 → 0.27.1 auto-recovery from Aborted: the new reconcile.recovery.from.aborted.onPodReady knob re-enqueues a CHI when a pod transitions NotReady → Ready while the CR sits in Aborted status. Default is retry (auto-recover on pod-Ready transition); set to none to opt out and preserve the pre-0.27.1 manual-intervention behaviour. (Renamed in 0.27.2 — see the 0.27.1 → 0.27.2 note below.)
0.27.1 → 0.27.2 recovery config key renamed (backward-incompatible): the auto-recovery config key reconcile.recovery.from.{aborted,completed} is renamed to reconcile.recovery.onStatus.{aborted,completed} in ClickHouseOperatorConfiguration. The from grouping level is removed; the per-status scopes (aborted, completed) and their action keys (onPodReady, onPodNotReady, onPodNotReadyThreshold) are otherwise unchanged. The operator silently ignores the obsolete from key (unknown config keys are dropped at load). Action required only if you explicitly set reconcile.recovery.from.aborted.onPodReady: none on 0.27.0/0.27.1 to disable auto-recovery of Aborted CHIs: after upgrade that setting is ignored, the accessor falls back to its default (retry), and Aborted auto-recovery is silently re-enabled. Re-apply it under the new path: reconcile.recovery.onStatus.aborted.onPodReady: none. (The completed scope — sustained-NotReady host recovery — is new in 0.27.2 and off by default, so no migration is needed for it.)
0.27.0 → 0.27.1 reconcile hooks (preview): new spec.reconcile.host.hooks and spec.reconcile.cluster.hooks blocks accept events: + sql: / http: / shell: actions. Only sql: is wired end-to-end in 0.27.1; http: and shell: currently emit a "not yet implemented" Fatal at validation time, so defer adopting those action types until the corresponding runners ship.
0.27.0 → 0.27.1 Keeper reference: the new spec.configuration.zookeeper.keeper field on ClickHouseInstallation lets you reference a ClickHouseKeeperInstallation by name; the operator resolves it to the matching ZK endpoint list automatically and re-reconciles dependent CHIs when the referenced CHK's endpoint set changes.
0.26.x → 0.27.0 CHK note: existing ClickHouse Keeper pods will be rolled (sequentially, gated by the startup probe) due to two operator-rendered keeper-config additions: an explicit <four_letter_word_white_list> (so the new ruok-based liveness probe keeps working even when users add restrictive keeper_server overrides) and the migration of the liveness probe itself from pgrep to a ruok/imok 4LW handshake. No data migration is required.
0.26.x → 0.27.0 metrics note: the default chop config now filters per-CPU OS metrics (metric.OSUserTimeCPU0..N, metric.CPUFrequencyMHz_*, etc.) from the metrics-exporter output. Dashboards or alerts that referenced these series will go silent after upgrade. To restore the prior behaviour, set excludeRegexp: [] in your ClickHouseOperatorConfiguration.
ClickHouse operator is deployed as Deployment Kubernetes resource (see: Operator Installation Guide for more details). Supplied clickhouse-operator-install-bundle.yaml contains the following deployment spec:
kind: Deployment
apiVersion: apps/v1
metadata:
name: clickhouse-operator
namespace: kube-system
labels:
clickhouse.altinity.com/chop: 0.17.0
app: clickhouse-operator
spec:
replicas: 1
selector:
matchLabels:
app: clickhouse-operator
template:
metadata:
labels:
app: clickhouse-operator
annotations:
prometheus.io/port: '8888'
prometheus.io/scrape: 'true'
clickhouse-operator-metrics/port: '9999'
clickhouse-operator-metrics/scrape: 'true'
spec:
serviceAccountName: clickhouse-operator
volumes:
- name: etc-clickhouse-operator-folder
configMap:
name: etc-clickhouse-operator-files
- name: etc-clickhouse-operator-confd-folder
configMap:
name: etc-clickhouse-operator-confd-files
- name: etc-clickhouse-operator-configd-folder
configMap:
name: etc-clickhouse-operator-configd-files
- name: etc-clickhouse-operator-templatesd-folder
configMap:
name: etc-clickhouse-operator-templatesd-files
- name: etc-clickhouse-operator-usersd-folder
configMap:
name: etc-clickhouse-operator-usersd-files
containers:
- name: clickhouse-operator
image: altinity/clickhouse-operator:0.17.2
imagePullPolicy: Always
volumeMounts:
- name: etc-clickhouse-operator-folder
mountPath: /etc/clickhouse-operator
- name: etc-clickhouse-operator-confd-folder
mountPath: /etc/clickhouse-operator/conf.d
- name: etc-clickhouse-operator-configd-folder
mountPath: /etc/clickhouse-operator/config.d
- name: etc-clickhouse-operator-templatesd-folder
mountPath: /etc/clickhouse-operator/templates.d
- name: etc-clickhouse-operator-usersd-folder
mountPath: /etc/clickhouse-operator/users.d
env:
# Pod-specific
- name: OPERATOR_POD_NODE_NAME
valueFrom:
fieldRef:
fieldPath: spec.nodeName
- name: OPERATOR_POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: OPERATOR_POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
- name: OPERATOR_POD_IP
valueFrom:
fieldRef:
fieldPath: status.podIP
- name: OPERATOR_POD_SERVICE_ACCOUNT
valueFrom:
fieldRef:
fieldPath: spec.serviceAccountName
# Container-specific
- name: OPERATOR_CONTAINER_CPU_REQUEST
valueFrom:
resourceFieldRef:
containerName: clickhouse-operator
resource: requests.cpu
- name: OPERATOR_CONTAINER_CPU_LIMIT
valueFrom:
resourceFieldRef:
containerName: clickhouse-operator
resource: limits.cpu
- name: OPERATOR_CONTAINER_MEM_REQUEST
valueFrom:
resourceFieldRef:
containerName: clickhouse-operator
resource: requests.memory
- name: OPERATOR_CONTAINER_MEM_LIMIT
valueFrom:
resourceFieldRef:
containerName: clickhouse-operator
resource: limits.memory
- name: metrics-exporter
image: altinity/metrics-exporter:0.17.0
imagePullPolicy: Always
volumeMounts:
- name: etc-clickhouse-operator-folder
mountPath: /etc/clickhouse-operator
- name: etc-clickhouse-operator-confd-folder
mountPath: /etc/clickhouse-operator/conf.d
- name: etc-clickhouse-operator-configd-folder
mountPath: /etc/clickhouse-operator/config.d
- name: etc-clickhouse-operator-templatesd-folder
mountPath: /etc/clickhouse-operator/templates.d
- name: etc-clickhouse-operator-usersd-folder
mountPath: /etc/clickhouse-operator/users.d
ports:
- containerPort: 8888
name: metrics
The latest available version is installed by default. If version changes, there are three ways to upgrade the operator:
- Delete existing deployment
kubectl delete deploy -n kube-system clickhouse-operator
- Upgrade Custom Resource Definitions to latest version
kubectl apply -f https://github.com/Altinity/clickhouse-operator/raw/master/deploy/operator/parts/crd.yaml
Current deployments can be listed using the following command:
$ kubectl describe --all-namespaces deployment -l app=clickhouse-operator
Name: clickhouse-operator
Namespace: kube-system
CreationTimestamp: Sat, 01 Jun 2019 23:44:46 +0300
Labels: app=clickhouse-operator
version=0.13.0
<...>
Pod Template:
Labels: app=clickhouse-operator
Service Account: clickhouse-operator
Containers:
clickhouse-operator:
Image: altinity/clickhouse-operator:0.17.0
metrics-exporter:
Image: altinity/metrics-exporter:0.17.0
<...>
Version is labeled and can be also displayed with the command:
$ kubectl get deployment --all-namespaces -L clickhouse.altinity.com/chop -l app=clickhouse-operator
NAMESPACE NAME UP-TO-DATE AVAILABLE AGE VERSION
kube-system clickhouse-operator 1 1 19h 0.17.0
If you want to update to the latest version, we can run following command:
$ kubectl apply -n kube-system -f https://github.com/Altinity/clickhouse-operator/raw/master/deploy/operator/clickhouse-operator-install-bundle.yaml
And then check upgrade status with:
$ kubectl get deployment --all-namespaces -L clickhouse.altinity.com/chop -l app=clickhouse-operator
NAMESPACE NAME READY UP-TO-DATE AVAILABLE AGE VERSION
kube-system clickhouse-operator 1/1 1 1 125d 0.18.2
$ kubectl describe --all-namespaces deployment -l app=clickhouse-operator
Name: clickhouse-operator
<...>
Pod Template:
Labels: app=clickhouse-operator
Service Account: clickhouse-operator
Containers:
clickhouse-operator:
Image: altinity/clickhouse-operator:0.18.2
metrics-exporter:
Image: altinity/metrics-exporter:0.18.2
<...>