ClickHouse Keeper missing readiness probe

[KJone1]

ClickHouse Keeper pods lack readiness probes, causing PDB to incorrectly report pods as healthy.

found this in pkg/model/chk/creator/probe.go:

  func (m *ProbeManager) CreateProbe(what interfaces.ProbeType, host *api.Host) *core.Probe {
  	switch what {
  	case interfaces.ProbeDefaultStartup:
  		return nil
  	case interfaces.ProbeDefaultLiveness:
  		return m.createDefaultLivenessProbe(host)
  	case interfaces.ProbeDefaultReadiness:
  		return nil
  		//return m.createDefaultReadinessProbe(host)  // COMMENTED OUT
  	}
  	panic("unknown probe type")
  }

keeper does support /ready endpoint defined here: https://github.com/ClickHouse/ClickHouse/blob/master/src/Server/KeeperReadinessHandler.cpp

[alex-zaitsev]

That was made intentional. When creating 3-node Keeper cluster one node after another, first node is never ready until it can build a quorum with a second one. But we should change the creation sequence maybe.

[dashashutosh24]

@alex-zaitsev I am using a CHK resource with the following config for replicas:

clusters:
    - layout:
        replicasCount: 3

I added the readinessProbe to podTemplates section following the exact same spec as given in

clickhouse-operator/pkg/model/chk/creator/probe.go

Line 75 in 0cf33cf

func (m *ProbeManager) createDefaultReadinessProbe(host *api.Host) *core.Probe {

and the pods are successfully coming up and forming the cluster.
Here is the spec of the statefulset after operator has rendered the CHK:

containers:
  - env:
    - name: CLICKHOUSE_DATA_DIR
      value: /var/lib/clickhouse-keeper
    image: clickhouse/clickhouse-keeper:25.3.5.42
    imagePullPolicy: IfNotPresent
    livenessProbe:
      exec:
        command:
        - bash
        - -xc
        - date && OK=$(exec 3<>/dev/tcp/127.0.0.1/2181; printf 'ruok' >&3; IFS=;
          tee <&3; exec 3<&-;);if [[ "${OK}" == "imok" ]]; then exit 0; else exit
          1; fi
      failureThreshold: 12
      initialDelaySeconds: 5
      periodSeconds: 5
      successThreshold: 1
      timeoutSeconds: 1
    name: clickhouse-keeper
    ports:
    - containerPort: 2181
      name: zk
      protocol: TCP
    - containerPort: 9444
      name: raft
      protocol: TCP
    readinessProbe:
      failureThreshold: 12
      httpGet:
        path: /ready
        port: 9182
        scheme: HTTP
      initialDelaySeconds: 5
      periodSeconds: 5
      successThreshold: 1
      timeoutSeconds: 1

And from settings side, these are the ones that I have set:

settings:
      keeper_server/coordination_settings/raft_logs_level: information
      keeper_server/four_letter_word_white_list: '*'
      keeper_server/log_storage_path: /var/lib/clickhouse-keeper/logs
      keeper_server/path: /var/lib/clickhouse-keeper
      keeper_server/snapshot_storage_path: /var/lib/clickhouse-keeper/snapshots
      keeper_server/tcp_port: "2181"
      listen_host: 0.0.0.0
      logger/console: "true"
      logger/level: trace
      prometheus/asynchronous_metrics: "true"
      prometheus/endpoint: /metrics
      prometheus/events: "true"
      prometheus/metrics: "true"
      prometheus/port: "7000"
      prometheus/status_info: "false"

Final list of pods:

chk-clickhouse-keeper-chkeeper-chkeeper-0-0-0               1/1     Running     0              9m11s
chk-clickhouse-keeper-chkeeper-chkeeper-0-1-0               1/1     Running     0              8m41s
chk-clickhouse-keeper-chkeeper-chkeeper-0-2-0               1/1     Running     0              8m23s

So, is there any reason for the removal of the probe by the operator that I am missing over here?

[alex-zaitsev]

Ok, that was actually a hack to overcome some Keeper weirdness. Enabled in upcoming 0.25.5

[alex-zaitsev]

62dc297

[alex-zaitsev]

https://github.com/Altinity/clickhouse-operator/releases/tag/release-0.25.5

[dashashutosh24]

@alex-zaitsev / @KJone1 I found one thing. While the readiness probe addition works when you upgrade existing installations with probes however the same fails on fresh installations as the pod itself keeps failing readiness check while trying to resolve the FQDN for other hosts in the cluster. I see logs such as the following for a fresh setup with readiness probes.

2025.10.28 09:41:26.967231 [ 65 ] {} <Information> RaftInstance: [ELECTION TIMEOUT] current role: candidate, log last term 8, state term 8, target p 1, my p 1, hb dead, pre-vote NOT done
2025.10.28 09:41:26.967248 [ 65 ] {} <Warning> RaftInstance: total 1 nodes (including this node) responded for pre-vote (term 8, live 0, dead 1), at least 2 nodes should respond. failure count 10
2025.10.28 09:41:26.967262 [ 65 ] {} <Information> RaftInstance: [PRE-VOTE INIT] my id 0, my role candidate, term 8, log idx 716359, log term 8, priority (target 1 / mine 1)
2025.10.28 09:41:26.967270 [ 65 ] {} <Warning> RaftInstance: failed to send prevote request: peer 2 (chk-clickhouse-keeper-chkeeper-chkeeper-0-2:9444) is busy
2025.10.28 09:41:26.967280 [ 65 ] {} <Warning> RaftInstance: failed to send prevote request: peer 1 (chk-clickhouse-keeper-chkeeper-chkeeper-0-1:9444) is busy
2025.10.28 09:41:28.845396 [ 64 ] {} <Warning> RaftInstance: Election timeout, initiate leader election
2025.10.28 09:41:28.845469 [ 64 ] {} <Information> RaftInstance: [PRIORITY] decay, target 1 -> 1, mine 1
2025.10.28 09:41:28.845483 [ 64 ] {} <Information> RaftInstance: [ELECTION TIMEOUT] current role: candidate, log last term 8, state term 8, target p 1, my p 1, hb dead, pre-vote NOT done
2025.10.28 09:41:28.845497 [ 64 ] {} <Warning> RaftInstance: total 1 nodes (including this node) responded for pre-vote (term 8, live 0, dead 1), at least 2 nodes should respond. failure count 11
2025.10.28 09:41:28.845521 [ 64 ] {} <Information> RaftInstance: [PRE-VOTE INIT] my id 0, my role candidate, term 8, log idx 716359, log term 8, priority (target 1 / mine 1)
2025.10.28 09:41:28.845529 [ 64 ] {} <Warning> RaftInstance: failed to send prevote request: peer 2 (chk-clickhouse-keeper-chkeeper-chkeeper-0-2:9444) is busy
2025.10.28 09:41:28.845539 [ 64 ] {} <Warning> RaftInstance: failed to send prevote request: peer 1 (chk-clickhouse-keeper-chkeeper-chkeeper-0-1:9444) is busy
2025.10.28 09:41:29.681296 [ 67 ] {} <Warning> RaftInstance: peer (2) response error: failed to resolve host chk-clickhouse-keeper-chkeeper-chkeeper-0-2 due to error 2, Host not found (non-authoritative), try again later
2025.10.28 09:41:30.015660 [ 66 ] {} <Warning> RaftInstance: Election timeout, initiate leader election
2025.10.28 09:41:30.015747 [ 66 ] {} <Information> RaftInstance: [PRIORITY] decay, target 1 -> 1, mine 1
2025.10.28 09:41:30.015767 [ 66 ] {} <Information> RaftInstance: [ELECTION TIMEOUT] current role: candidate, log last term 8, state term 8, target p 1, my p 1, hb dead, pre-vote NOT done
2025.10.28 09:41:30.015785 [ 66 ] {} <Information> RaftInstance: reset RPC client for peer 2
2025.10.28 09:41:30.015909 [ 66 ] {} <Warning> RaftInstance: total 1 nodes (including this node) responded for pre-vote (term 8, live 0, dead 1), at least 2 nodes should respond. failure count 12
2025.10.28 09:41:30.015929 [ 66 ] {} <Information> RaftInstance: [PRE-VOTE INIT] my id 0, my role candidate, term 8, log idx 716359, log term 8, priority (target 1 / mine 1)
2025.10.28 09:41:30.015948 [ 66 ] {} <Warning> RaftInstance: failed to send prevote request: peer 1 (chk-clickhouse-keeper-chkeeper-chkeeper-0-1:9444) is busy

As a result, the first replica itself is stuck and cluster doesn't progress towards completion. But while upgrading, since hosts already exist, I think that is why cluster formation succeeds.