1. Prerequisites
- The following article was written with Bright Cluster Manager 10 and 11 in mind but should work the same for version 9.2 as well.
- For BCM 9.0 and 9.1, please refer to the old instructions in this version of the KB article:
https://kb.brightcomputing.com/knowledge-base/etcd-membership-reconfiguration-in-bright-9-0/
- For BCM 9.0 and 9.1, please refer to the old instructions in this version of the KB article:
- The minimum required version of Etcd should be at least 3.4.4 (earlier versions, such as 3.4.3 contain serious bugs).
- BCM 9.2 and up require the script
cm-etcd-managesee instructions below how to download it. - BCM 11 and up additionally require the script
cm-kubeadm-manage, see instructions below how to download it.
1.1. Downloading the helper scripts
These downloads will not be necessary if the version of BCM is either >= 10.25.06 or >= 11.25.05.
# BCM 10+ wget -O /cm/local/apps/cmd/scripts/cm-etcd-manage https://support2.brightcomputing.com/etcd/cm-etcd-manage wget -O /cm/local/apps/cmd/scripts/cm-kubeadm-manage https://support2.brightcomputing.com/etcd/cm-kubeadm-manage chmod +x /cm/local/apps/cmd/scripts/cm-etcd-manage chmod +x /cm/local/apps/cmd/scripts/cm-kubeadm-manage # BCM 9.2 (only cm-etcd-manage is needed) wget -O /cm/local/apps/cmd/scripts/cm-etcd-manage https://support2.brightcomputing.com/etcd/cm-etcd-manage chmod +x /cm/local/apps/cmd/scripts/cm-etcd-manage
1.2. Example usage
We assume two aliases are in place in order to keep the KB article output more readable:
alias cm-etcd-manage='/cm/local/apps/cmd/scripts/cm-etcd-manage' alias cm-kubeadm-manage='/cm/local/apps/cmd/scripts/cm-kubeadm-manage'
These are the options for the cm-etcd-manage tool:
# cm-etcd-manage --help
usage: cm-etcd-manage [-h] --kube-cluster KUBE_CLUSTER {status,remove,add,add-learner,promote,move-leader,backup,restore} ...
...
- The
--kube-clusterparameter is mandatory, and actions (status, remove, etc.) may require additional parameters. - In this KB article we will assume the kube cluster we are dealing with has the label/name
default. The list of kube clusters managed by BCM can be listed bycmshas follows:cmsh -c 'kubernetes list'.
These are the options for the cm-kubeadm-manage tool:
# cm-kubeadm-manage --help
usage: cm-kubeadm-manage [-h] --kube-cluster KUBE_CLUSTER
{status,update_configmap,update,update_apiserver,update_controller_manager,update_scheduler,upda
te_apiserver_cert,update_certs,check_certs}
...
2. Background
Crucially, a majority of nodes have to agree to elect a new leader (2/3, 4/6, etc.), and if a majority can’t be reached, the entire cluster will be unavailable. What this means in practice is that etcd will remain available as long as a majority of nodes is online.
Source: here
In Etcd terminology a node is also often referred to as a member. In this article we will try to talk about members and use the term node for the underlying machine.
This means a three-member cluster can tolerate one broken member. And a five-member cluster can tolerate two broken members.
To ensure a majority of members in a healthy state at all times, it is recommended to remove or add always one member at a time.
3. Checking Etcd cluster status
In the below example we see a three member Etcd cluster where node002 is the elected leader (see the last column in the first printed table)
The columns showing the raft index for all three show the numbers are really close, suggesting they are well in-sync. The second table also shows the endpoint health in the second column. “true” meaning health is okay. In case something is wrong, it will not say “true” and the error column will contain more information.
# cm-etcd-manage --kube-cluster=default status ... 2025-05-06 10:01:53,490 - cm-etcd-manage - INFO - Etcd cluster status: +----------------------------------------+-----------+----------+-------------+------------+--------------------+---------------+---------------------+---------------------+------------+-------------+-------------+ | node | version | dbSize | raftIndex | raftTerm | raftAppliedIndex | dbSizeInUse | cluster_id | member_id | revision | raft_term | is_leader | +========================================+===========+==========+=============+============+====================+===============+=====================+=====================+============+=============+=============+ | ci-tmp-100-u2204-creative-porch-060856 | 3.5.15 | 36868096 | 16732 | 3 | 16732 | 10600448 | 9258346023728438660 | 2200820651967761369 | 15018 | 3 | false | +----------------------------------------+-----------+----------+-------------+------------+--------------------+---------------+---------------------+---------------------+------------+-------------+-------------+ | node001 | 3.5.15 | 36220928 | 16732 | 3 | 16732 | 10575872 | 9258346023728438660 | 5346736740293606848 | 15018 | 3 | false | +----------------------------------------+-----------+----------+-------------+------------+--------------------+---------------+---------------------+---------------------+------------+-------------+-------------+ | node002 | 3.5.15 | 36253696 | 16732 | 3 | 16732 | 10612736 | 9258346023728438660 | 1211154242112782154 | 15018 | 3 | true | +----------------------------------------+-----------+----------+-------------+------------+--------------------+---------------+---------------------+---------------------+------------+-------------+-------------+ 2025-05-06 10:01:53,494 - cm-etcd-manage - INFO - Etcd cluster health: 2025-05-06 10:01:53,495 - cm-etcd-manage - DEBUG - Executing: . /etc/profile.d/modules.sh ; module load etcd && etcdctl endpoint health --cluster --write-out=table +-----------------------------+--------+-------------+-------+ | ENDPOINT | HEALTH | TOOK | ERROR | +-----------------------------+--------+-------------+-------+ | https://10.141.0.2:2379 | true | 15.918309ms | | | https://10.141.255.254:2379 | true | 22.516309ms | | | https://10.141.0.1:2379 | true | 34.635825ms | | +-----------------------------+--------+-------------+-------+
Keeping an eye on the status and health of Etcd using the above command before and after making changes to the cluster is recommended. It’s important to keep always a majority of nodes (1 in case of 1 member, 2 in case of 3 members, 3/5, 4/6, etc.) up & healthy.
4. Create a backup (snapshot)
Creating a snapshot can be taxing on the database, so we recommended to take a snapshot from the leader and not for all members at once. After the snapshot, double check with the status command to see if all members are still healthy, and that there is no big discrepancy between raft indexes.
# cm-etcd-manage --kube-cluster default backup /cm/shared/etcd-backups 2025-05-01 15:02:44,198 - cm-etcd-manage - INFO - Creating backup as /cm/shared/etcd-backups/etcd-backup-node002-010903-2025-05-01-15-02-44.db
We can find our snapshot in the provided target directory:
# ls -al /cm/shared/etcd-backups/ total 6000 drwxr-xr-x 2 root root 85 May 1 15:02 . drwxr-xr-x 9 root root 112 May 1 15:02 .. -rw------- 1 root root 6139936 May 1 15:02 etcd-backup-node002-2025-05-01-15-02-44.db
(To take a snapshot of all the members anyway, use: cm-etcd-manage --kube-cluster default backup /cm/shared/etcd-backups --all-nodes.)
5. Adding a new member
We will assume the cm-etcd package is present on the node (the script will check for requirement). If it’s not installed on the node, please install the package in the relevant software image, and issue an imageupdate (e.g. in cmsh‘s device mode using the imageupdate command).
# cm-etcd-manage --kube-cluster=default add node003 ... 2025-05-06 10:13:12,993 - cm-etcd-manage - INFO - Etcd learner node node003 is ready in cluster default 2025-05-06 10:13:14,292 - cm-etcd-manage - INFO - Etcd cluster status: +----------------------------------------+-----------+----------+-------------+------------+--------------------+---------------+---------------------+----------------------+------------+-------------+-------------+--------------+ | node | version | dbSize | raftIndex | raftTerm | raftAppliedIndex | dbSizeInUse | cluster_id | member_id | revision | raft_term | is_leader | is_learner | +========================================+===========+==========+=============+============+====================+===============+=====================+======================+============+=============+=============+==============+ | ci-tmp-100-u2204-creative-porch-060856 | 3.5.15 | 36868096 | 20392 | 4 | 20392 | 10792960 | 9258346023728438660 | 2200820651967761369 | 18259 | 4 | false | | +----------------------------------------+-----------+----------+-------------+------------+--------------------+---------------+---------------------+----------------------+------------+-------------+-------------+--------------+ | node001 | 3.5.15 | 36220928 | 20392 | 4 | 20392 | 10813440 | 9258346023728438660 | 5346736740293606848 | 18259 | 4 | true | | +----------------------------------------+-----------+----------+-------------+------------+--------------------+---------------+---------------------+----------------------+------------+-------------+-------------+--------------+ | node002 | 3.5.15 | 36253696 | 20392 | 4 | 20392 | 10821632 | 9258346023728438660 | 1211154242112782154 | 18259 | 4 | false | | +----------------------------------------+-----------+----------+-------------+------------+--------------------+---------------+---------------------+----------------------+------------+-------------+-------------+--------------+ | node003 | 3.5.15 | 36253696 | 20392 | 4 | 20392 | 10825728 | 9258346023728438660 | 17442037826836741677 | 18259 | 4 | false | true | +----------------------------------------+-----------+----------+-------------+------------+--------------------+---------------+---------------------+----------------------+------------+-------------+-------------+--------------+ ... 2025-05-06 10:13:27,120 - cm-etcd-manage - INFO - Node node003 is now a regular member in cluster default 2025-05-06 10:13:27,220 - cm-etcd-manage - INFO - Marking node node003 as a datanode in cluster default 2025-05-06 10:13:27,220 - cm-etcd-manage - INFO - Node node003 is now a datanode ... 2025-05-06 10:13:28,523 - cm-etcd-manage - INFO - Etcd cluster status: +----------------------------------------+-----------+----------+-------------+------------+--------------------+---------------+---------------------+----------------------+------------+-------------+-------------+ | node | version | dbSize | raftIndex | raftTerm | raftAppliedIndex | dbSizeInUse | cluster_id | member_id | revision | raft_term | is_leader | +========================================+===========+==========+=============+============+====================+===============+=====================+======================+============+=============+=============+ | ci-tmp-100-u2204-creative-porch-060856 | 3.5.15 | 36868096 | 20468 | 4 | 20468 | 10842112 | 9258346023728438660 | 2200820651967761369 | 18324 | 4 | false | +----------------------------------------+-----------+----------+-------------+------------+--------------------+---------------+---------------------+----------------------+------------+-------------+-------------+ | node001 | 3.5.15 | 36220928 | 20468 | 4 | 20468 | 10858496 | 9258346023728438660 | 5346736740293606848 | 18324 | 4 | true | +----------------------------------------+-----------+----------+-------------+------------+--------------------+---------------+---------------------+----------------------+------------+-------------+-------------+ | node002 | 3.5.15 | 36253696 | 20468 | 4 | 20468 | 10874880 | 9258346023728438660 | 1211154242112782154 | 18324 | 4 | false | +----------------------------------------+-----------+----------+-------------+------------+--------------------+---------------+---------------------+----------------------+------------+-------------+-------------+ | node003 | 3.5.15 | 36253696 | 20468 | 4 | 20468 | 10870784 | 9258346023728438660 | 17442037826836741677 | 18324 | 4 | false | +----------------------------------------+-----------+----------+-------------+------------+--------------------+---------------+---------------------+----------------------+------------+-------------+-------------+ 2025-05-06 10:13:28,527 - cm-etcd-manage - INFO - Etcd cluster health: 2025-05-06 10:13:28,527 - cm-etcd-manage - DEBUG - Executing: . /etc/profile.d/modules.sh ; module load etcd && etcdctl endpoint health --cluster --write-out=table +-----------------------------+--------+-------------+-------+ | ENDPOINT | HEALTH | TOOK | ERROR | +-----------------------------+--------+-------------+-------+ | https://10.141.0.1:2379 | true | 15.78864ms | | | https://10.141.0.2:2379 | true | 26.585208ms | | | https://10.141.0.3:2379 | true | 27.156311ms | | | https://10.141.255.254:2379 | true | 54.542171ms | | +-----------------------------+--------+-------------+-------+
The above output has been abbreviated for readability, two things to point out is that after the new node003 has been added it first is added as a learner. This is also shown in the last column of the first printed table.
After node003 was added as a learner successfully the node was promoted to leader, and after that two more tables were printed, these tables show the cluster is healthy.
The script will have also marked the device as a datanode:
# cmsh [ci-tmp-100-u2204-creative-porch-060856]% device use node003 [ci-tmp-100-u2204-creative-porch-060856->device[node003]]% get datanode yes
If the goal is to end up with a five member Etcd cluster; it makes sense to first expand the cluster to five, and then to continue with the next section to update Kube API server configuration.
6. Updating Kubernetes API server configurations
This section is not necessary for BCM 9.2 clusters, since in that case BCM fully manages the kube-apiserver configuration, and deals with required restarts. In BCM 10 and higher we delegate this to kubeadm.
As an example purposes, we are dealing with two control-plane nodes:
# kubectl get nodes NAME STATUS ROLES AGE VERSION headnode Ready control-plane,master 4h23m v1.31.8 node001 Ready control-plane,master,worker 4h25m v1.31.8 node002 Ready worker 4h24m v1.31.8 node003 Ready worker 4h24m v1.31.8 node004 Ready worker 4h24m v1.31.8 node005 Ready worker 4h24m v1.31.8 node006 Ready worker 4h24m v1.31.8
We can see that in the Kubernetes API server manifest file, on both control-plane nodes, the new Etcd member is not automatically passed to the Kube API server. We need kubeadm to regenerate the manifest file on the two nodes.
# pdsh -w headnode,node001 grep etcd-servers /etc/kubernetes/manifests/kube-apiserver.yaml headnode: - --etcd-servers=https://10.141.255.254:2379,https://10.141.0.1:2379,https://10.141.0.2:2379 node001: - --etcd-servers=https://10.141.255.254:2379,https://10.141.0.1:2379,https://10.141.0.2:2379
The previous section expanded the Etcd cluster to be four nodes. First we have to manually update the config map that will used be used by kubeadm. We will use the cm-kubeadm-config helper for that, and we can start by issuing a status action first.
# cm-kubeadm-manage --kube-cluster=default status ... 2025-05-06 13:51:51,970 - cm-kubeadm-manage - INFO - Kubernetes kube-apiserver control-plane pods: NAME READY STATUS RESTARTS AGE kube-apiserver-headnode 1/1 Running 0 4h27m kube-apiserver-node001 1/1 Running 0 4h31m 2025-05-06 13:51:51,971 - cm-kubeadm-manage - INFO - Kubernetes kube-controller-manager control-plane pods: NAME READY STATUS RESTARTS AGE kube-controller-manager-headnode 1/1 Running 0 4h31m kube-controller-manager-node001 1/1 Running 1 (19m ago) 4h32m 2025-05-06 13:51:51,971 - cm-kubeadm-manage - INFO - Kubernetes kube-scheduler control-plane pods: NAME READY STATUS RESTARTS AGE kube-scheduler-headnode 1/1 Running 0 4h31m kube-scheduler-node001 1/1 Running 2 (18m ago) 4h32m 2025-05-06 13:51:51,971 - cm-kubeadm-manage - INFO - Kubeadm-init files on master nodes: -rw------- 1 root root 1.1K May 6 09:18 /root/.kube/kubeadm-init-default.yaml -rw------- 1 root root 1.1K May 6 09:18 /root/.kube/kubeadm-init-default.yaml 2025-05-06 13:51:51,972 - cm-kubeadm-manage - INFO - SANs from the kubeadm config: - active - master - localhost - 10.141.255.254 - 10.141.0.1 2025-05-06 13:51:51,980 - cm-kubeadm-manage - INFO - List of Current Etcd Members: - https://10.141.255.254:2379 - https://10.141.0.3:2379 - https://10.141.0.1:2379 - https://10.141.0.2:2379 2025-05-06 13:51:51,981 - cm-kubeadm-manage - INFO - List of Etcd Members in Config Map: - https://10.141.255.254:2379 - https://10.141.0.1:2379 - https://10.141.0.2:2379 2025-05-06 13:51:51,981 - cm-kubeadm-manage - WARNING - Missing Etcd Members in Config Map: https://10.141.0.3:2379 (use: 'update_configmap' to fix) 2025-05-06 13:51:51,981 - cm-kubeadm-manage - INFO - No extra Etcd Members in Config Map 2025-05-06 13:51:51,981 - cm-kubeadm-manage - WARNING - Missing Etcd Members in node001:/etc/kubernetes/manifests/kube-apiserver.yaml: https://10.141.0.3:2379 (use: 'update_configmap' and 'update_apiserver node001' to fix) 2025-05-06 13:51:51,981 - cm-kubeadm-manage - INFO - No extra Etcd Members in node001:/etc/kubernetes/manifests/kube-apiserver.yaml 2025-05-06 13:51:51,981 - cm-kubeadm-manage - WARNING - Missing Etcd Members in headnode:/etc/kubernetes/manifests/kube-apiserver.yaml: https://10.141.0.3:2379 (use: 'update_configmap' and 'update_apiserver ci-tmp-100-u2204-creative-porch-060856' to fix) 2025-05-06 13:51:51,981 - cm-kubeadm-manage - INFO - No extra Etcd Members in headnode:/etc/kubernetes/manifests/kube-apiserver.yaml
The output command summarized near the end the output that certain Etcd members were missing in the configmap, and the manifest files. We can use the update_configmap action (as the output also suggested) to deal with the configmap first.
# cm-kubeadm-manage --kube-cluster=default update_configmap ... 2025-05-06 13:53:33,024 - cm-kubeadm-manage - DEBUG - Executing: kubectl --kubeconfig=/root/.kube/config-default apply -f /tmp/tmpaexwu0bl.yaml 2025-05-06 13:53:33,400 - cm-kubeadm-manage - INFO - Successfully updated kubeadm config 2025-05-06 13:53:33,453 - cm-kubeadm-manage - INFO - Writing kubeadm-init file to headnode 2025-05-06 13:53:33,943 - cm-kubeadm-manage - INFO - Successfully wrote /root/.kube/kubeadm-init-default.yaml on headnode 2025-05-06 13:53:33,959 - cm-kubeadm-manage - INFO - Writing kubeadm-init file to node001 2025-05-06 13:53:34,422 - cm-kubeadm-manage - INFO - Successfully wrote /root/.kube/kubeadm-init-default.yaml on node001
Now we repeat the status action.
# cm-kubeadm-manage --kube-cluster=default status ... 2025-05-06 13:55:10,483 - cm-kubeadm-manage - INFO - No missing Etcd Members in Config Map 2025-05-06 13:55:10,483 - cm-kubeadm-manage - INFO - No extra Etcd Members in Config Map 2025-05-06 13:55:10,483 - cm-kubeadm-manage - WARNING - Missing Etcd Members in node001:/etc/kubernetes/manifests/kube-apiserver.yaml: https://10.141.0.3:2379 (use: 'update_apiserver node001' to fix) 2025-05-06 13:55:10,483 - cm-kubeadm-manage - INFO - No extra Etcd Members in node001:/etc/kubernetes/manifests/kube-apiserver.yaml 2025-05-06 13:55:10,483 - cm-kubeadm-manage - WARNING - Missing Etcd Members in headnode:/etc/kubernetes/manifests/kube-apiserver.yaml: https://10.141.0.3:2379 (use: 'update_apiserver headnode' to fix) 2025-05-06 13:55:10,483 - cm-kubeadm-manage - INFO - No extra Etcd Members in headnode:/etc/kubernetes/manifests/kube-apiserver.yaml
We will deal with regeneration of the kube-apiserver manifest file + restarting the pod, one control-plane node at a time.
# cm-kubeadm-manage --kube-cluster=default update_apiserver headnode 2025-05-06 13:56:24,571 - cm-kubeadm-manage - DEBUG - Executing: kubeadm init phase control-plane apiserver --config /root/.kube/kubeadm-init-default.yaml --v=5 [control-plane] Creating static Pod manifest for "kube-apiserver" ... kube-apiserver Pod is restarted ... # cm-kubeadm-manage --kube-cluster=default update_apiserver node001 [control-plane] Creating static Pod manifest for "kube-apiserver" ... etc...
After the above two commands both Kube API servers should now be aware of the four Etcd members.
7. Removing an Etcd member
We start again with checking the status.
# cm-etcd-manage --kube-cluster=default status ... 2025-05-06 14:02:36,886 - cm-etcd-manage - INFO - Etcd cluster status: +----------+-----------+----------+-------------+------------+--------------------+---------------+---------------------+---------------------+------------+-------------+-------------+ | node | version | dbSize | raftIndex | raftTerm | raftAppliedIndex | dbSizeInUse | cluster_id | member_id | revision | raft_term | is_leader | +==========+===========+==========+=============+============+====================+===============+=====================+=====================+============+=============+=============+ | headnode | 3.5.15 | 36868096 | 95303 | 18 | 95303 | 10719232 | 9258346023728438660 | 2200820651967761369 | 84755 | 18 | true | +----------+-----------+----------+-------------+------------+--------------------+---------------+---------------------+---------------------+------------+-------------+-------------+ | node001 | 3.5.15 | 36220928 | 95303 | 18 | 95303 | 10752000 | 9258346023728438660 | 5346736740293606848 | 84755 | 18 | false | +----------+-----------+----------+-------------+------------+--------------------+---------------+---------------------+---------------------+------------+-------------+-------------+ | node002 | 3.5.15 | 36253696 | 95303 | 18 | 95303 | 10772480 | 9258346023728438660 | 1211154242112782154 | 84755 | 18 | false | +----------+-----------+----------+-------------+------------+--------------------+---------------+---------------------+---------------------+------------+-------------+-------------+ | node003 | 3.5.15 | 36868096 | 95303 | 18 | 95303 | 10735616 | 9258346023728438660 | 6140163318520876536 | 84755 | 18 | false | +----------+-----------+----------+-------------+------------+--------------------+---------------+---------------------+---------------------+------------+-------------+-------------+ 2025-05-06 14:02:36,889 - cm-etcd-manage - INFO - Etcd cluster health: 2025-05-06 14:02:36,890 - cm-etcd-manage - DEBUG - Executing: . /etc/profile.d/modules.sh ; module load etcd && etcdctl endpoint health --cluster --write-out=table +-----------------------------+--------+--------------+-------+ | ENDPOINT | HEALTH | TOOK | ERROR | +-----------------------------+--------+--------------+-------+ | https://10.141.0.2:2379 | true | 15.192343ms | | | https://10.141.255.254:2379 | true | 212.756948ms | | | https://10.141.0.3:2379 | true | 214.343101ms | | | https://10.141.0.1:2379 | true | 215.781094ms | | +-----------------------------+--------+--------------+-------+
In the above case, removing the headnode will throw an error (the script won’t accept removal of headnode), and prints an error:
FAQ: 2025-05-01 15:27:45,239 - cm-etcd-manage - ERROR - Please use the move-leader command to move the leader to another node
In order to fix that, we have to move the leader to one of the others, e.g., to node002 with: cm-etcd-manage --kube-cluster=default move-leader node002.
We are going to delete node003 (10.141.0.3), so we don’t have to move the leader node in our case.
# cm-etcd-manage --kube-cluster=default remove node003 ... 2025-05-06 14:05:26,197 - cm-etcd-manage - INFO - Etcd service on node node003 is still running 2025-05-06 14:05:36,307 - cm-etcd-manage - DEBUG - Executing: systemctl status etcd 2025-05-06 14:05:36,597 - cm-etcd-manage - INFO - Etcd service on node node003 is not running 2025-05-06 14:05:36,894 - cm-etcd-manage - INFO - Removing left over CA file /etc/kubernetes/pki/default/etcd/ca.crt 2025-05-06 14:05:37,007 - cm-etcd-manage - DEBUG - Executing: rm -rfv /var/lib/etcd/* 2025-05-06 14:05:37,115 - cm-etcd-manage - INFO - Removed left over /var/lib/etcd/* on node node003 2025-05-06 14:05:37,116 - cm-etcd-manage - DEBUG - removed '/var/lib/etcd/member/snap/db' removed '/var/lib/etcd/member/snap/0000000000000012-000000000001525f.snap' removed '/var/lib/etcd/member/snap/0000000000000012-00000000000165e8.snap' removed directory '/var/lib/etcd/member/snap' removed '/var/lib/etcd/member/wal/0000000000000000-0000000000000000.wal' removed directory '/var/lib/etcd/member/wal' removed directory '/var/lib/etcd/member' ... 2025-05-06 14:05:51,449 - cm-etcd-manage - INFO - Etcd cluster status: +----------+-----------+----------+-------------+------------+--------------------+---------------+---------------------+---------------------+------------+-------------+-------------+ | node | version | dbSize | raftIndex | raftTerm | raftAppliedIndex | dbSizeInUse | cluster_id | member_id | revision | raft_term | is_leader | +==========+===========+==========+=============+============+====================+===============+=====================+=====================+============+=============+=============+ | headnode | 3.5.15 | 36868096 | 96336 | 23 | 96336 | 10113024 | 9258346023728438660 | 2200820651967761369 | 85658 | 23 | true | +----------+-----------+----------+-------------+------------+--------------------+---------------+---------------------+---------------------+------------+-------------+-------------+ | node001 | 3.5.15 | 36220928 | 96336 | 23 | 96336 | 10153984 | 9258346023728438660 | 5346736740293606848 | 85658 | 23 | false | +----------+-----------+----------+-------------+------------+--------------------+---------------+---------------------+---------------------+------------+-------------+-------------+ | node002 | 3.5.15 | 36253696 | 96336 | 23 | 96336 | 10162176 | 9258346023728438660 | 1211154242112782154 | 85658 | 23 | false | +----------+-----------+----------+-------------+------------+--------------------+---------------+---------------------+---------------------+------------+-------------+-------------+ 2025-05-06 14:05:51,452 - cm-etcd-manage - INFO - Etcd cluster health: 2025-05-06 14:05:51,452 - cm-etcd-manage - DEBUG - Executing: . /etc/profile.d/modules.sh ; module load etcd && etcdctl endpoint health --cluster --write-out=table +-----------------------------+--------+-------------+-------+ | ENDPOINT | HEALTH | TOOK | ERROR | +-----------------------------+--------+-------------+-------+ | https://10.141.255.254:2379 | true | 12.85361ms | | | https://10.141.0.2:2379 | true | 22.60545ms | | | https://10.141.0.1:2379 | true | 27.646804ms | | +-----------------------------+--------+-------------+-------+
Please note that now that we removed one Etcd member (node003, which is 10.141.0.3), for BCM 10+, we again need to make sure the control-plane nodes, that are running the Kubernetes API servers are updated with the proper configuration.
The below output shows that they are still configured with the now removed endpoint.
# pdsh -w headnode,node001 grep etcd-servers /etc/kubernetes/manifests/kube-apiserver.yaml headnode: - --etcd-servers=https://10.141.255.254:2379,https://10.141.0.1:2379,https://10.141.0.2:2379,https://10.141.0.3:2379 node001: - --etcd-servers=https://10.141.255.254:2379,https://10.141.0.1:2379,https://10.141.0.2:2379,https://10.141.0.3:2379
We have to repeat the steps in Section 6, which in summary for this example are the following calls to cm-kubeadm-manage.
# cm-kubeadm-manage --kube-cluster=default update_configmap ... # cm-kubeadm-manage --kube-cluster=default update_apiserver headnode ... # cm-kubeadm-manage --kube-cluster=default update_apiserver node001
After this the Kube API servers no longer try to connect with the now removed member.
8. Common scenarios
We will explicitly go over the following common scenarios:
- 8.1. Extend a single-member Etcd cluster to three members.
- 8.2. Replace a member, due to unexpected hardware failure.
- 8.3. Migrate one of the members to another node.
- 8.4. Restore a previously made backup (disaster recovery)
Please know that at all times we can query the Etcd cluster status using: cm-etcd-manage --kube-cluster=<cluster> status. This will be useful to keep track of the health of the endpoints and status (who is a learner and/or leader, etc.)
8.1. Scenario: Extend a single-member Etcd cluster to three members.
# Take a snapshot first (section 4) cm-etcd-manage --kube-cluster=<cluster> backup <shared_storage_path> # Add two new members (twice section 5) cm-etcd-manage --kube-cluster=<cluster> add <new_node_1> cm-etcd-manage --kube-cluster=<cluster> add <new_node_2> # Update Kube API servers configuration (section 6) cm-kubeadm-manage --kube-cluster=<cluster> update_configmap # For each control-plane node: cm-kubeadm-manage --kube-cluster=<cluster> update_apiserver <node>
Please be careful the other way around: going back from three to one node is not as trivial since Etcd will get into trouble for not maintaining a majority when you downsize from 2 to 1 Etcd members. In that case, taking a snapshot and restoring it in only one node is likely the only way to do it.
8.2. Scenario: Replace a member, due to unexpected hardware failure.
In below example the broken_etcd_node, may be the same node we add as a new_etcd_node below.
(If the replacement node is a different node, it is also possible to add the replacement node first, and then remove the broken member. Basically the same as the scenario in 8.3 in that case)
# Take a snapshot first (section 4) cm-etcd-manage --kube-cluster=<cluster> backup <shared_storage_path> # Remove the broken member (section 7) cm-etcd-manage --kube-cluster=<cluster> remove <broken_etcd_node> # Add back the new member (section 5) cm-etcd-manage --kube-cluster=<cluster> add <new_etcd_node> # Update Kube API servers configuration (section 6) cm-kubeadm-manage --kube-cluster=<cluster> update_configmap # For each control-plane node: cm-kubeadm-manage --kube-cluster=<cluster> update_apiserver <node>
8.3. Scenario: Migrate one of the members to another node.
Simplest is just to add an additional member on the target node, and then remove the old one.
# Take a snapshot first (section 4) cm-etcd-manage --kube-cluster=<cluster> backup <shared_storage_path> # Add a new member at target location (section 5) cm-etcd-manage --kube-cluster=<cluster> add <new_node> # Remove the old location member (section 7) cm-etcd-manage --kube-cluster=<cluster> remove <old_node> # Update Kube API servers configuration (section 6) cm-kubeadm-manage --kube-cluster=<cluster> update_configmap # For each control-plane node: cm-kubeadm-manage --kube-cluster=<cluster> update_apiserver <node>
8.4. Scenario: Restore a previously made backup (disaster recovery)
This assumes there is a snapshot file available to restore. Please refer to Section 4 for the steps in details.
The best way to restore the Etcd cluster is to unassign the Etcd::Host role from all the nodes, and wait for BCM to stop all the Etcd services accross the cluster.
# cmsh [headnode]% configurationoverlay [headnode->configurationoverlay]% use kube-default-etcd [headnode->configurationoverlay[kube-default-etcd]]% clear nodes [headnode->configurationoverlay*[kube-default-etcd*]]% show Parameter Value -------------------------------- ------------------------------------------------ Name kube-default-etcd Revision All head nodes no Priority 500 Nodes Categories Roles Etcd::Host Customizations <0 in submode> [headnode->configurationoverlay*[kube-default-etcd*]]% commit [headnode->configurationoverlay[kube-default-etcd]]% Tue May 6 14:37:58 2025 [notice] headnode: Service etcd was stopped Tue May 6 14:38:02 2025 [notice] node002: Service etcd was stopped Tue May 6 14:38:07 2025 [notice] node001: Service etcd was stopped [headnode->configurationoverlay[kube-default-etcd]]%
Now that Etcd is not running anymore, we will choose a node to restore the backup to. In our case, we will pick the node002 to illustrate the backup can be restored everywhere. The leader was node001 at the time of the backup, we will restore this file:
root@headnode:~# ls -alh /cm/shared/etcd-backup-node001-2025-05-06-14-37-39.db -rw------- 1 root root 35M May 6 14:37 /cm/shared/etcd-backup-node001-2025-05-06-14-37-39.db
And we will restore the backup on node002 first.
There is one thing we have to ensure before we do the backup (the script will complain about it otherwise), is remove old data from /var/lib/etcd, like this:
# ssh node002 'rm -rfv /var/lib/etcd/*' removed '/var/lib/etcd/member/snap/db' removed '/var/lib/etcd/member/snap/000000000000000c-0000000000014c1a.snap' removed '/var/lib/etcd/member/snap/0000000000000012-0000000000015fa3.snap' removed '/var/lib/etcd/member/snap/0000000000000012-000000000001732c.snap' removed '/var/lib/etcd/member/snap/0000000000000017-00000000000186b5.snap' removed '/var/lib/etcd/member/snap/0000000000000019-0000000000019a3e.snap' removed directory '/var/lib/etcd/member/snap' removed '/var/lib/etcd/member/wal/0000000000000000-0000000000000000.wal' removed '/var/lib/etcd/member/wal/0000000000000001-0000000000009d2a.wal' removed directory '/var/lib/etcd/member/wal' removed directory '/var/lib/etcd/member'
Next we will do the restore.
# cm-etcd-manage --kube-cluster=default restore node002 /cm/shared/etcd-backup-node001-2025-05-06-14-37-39.db ... 2025-05-06 14:42:34,551 - cm-etcd-manage - DEBUG - Executing: /cm/local/apps/etcd/current/bin/etcdctl snapshot restore /cm/shared/etcd-backup-node001-2025-05-06-14-37-39.db --data-dir=/var/lib/etcd --name=n ode002 --initial-cluster=node002=https://10.141.0.2:2380 --initial-cluster-token=etcd-cluster-1 --initial-advertise-peer-urls=https://10.141.0.2:2380 ... 2025-05-06 14:42:35,844 - cm-etcd-manage - INFO - Etcd cluster default should be restored.. 2025-05-06 14:42:37,877 - cm-etcd-manage - INFO - Node node002 is now in etcd overlay kube-default-etcd
We will check with a status command that everything went fine.
# cm-etcd-manage --kube-cluster=default status ... 2025-05-06 14:43:41,822 - cm-etcd-manage - INFO - Etcd cluster status: +---------+-----------+----------+-------------+------------+--------------------+---------------+----------------------+----------------------+------------+-------------+-------------+ | node | version | dbSize | raftIndex | raftTerm | raftAppliedIndex | dbSizeInUse | cluster_id | member_id | revision | raft_term | is_leader | +=========+===========+==========+=============+============+====================+===============+======================+======================+============+=============+=============+ | node002 | 3.5.15 | 36220928 | 7 | 2 | 7 | 10530816 | 10999330578793104302 | 12480638411775839385 | 94714 | 2 | true | +---------+-----------+----------+-------------+------------+--------------------+---------------+----------------------+----------------------+------------+-------------+-------------+ 2025-05-06 14:43:41,827 - cm-etcd-manage - INFO - Etcd cluster health: 2025-05-06 14:43:41,827 - cm-etcd-manage - DEBUG - Executing: . /etc/profile.d/modules.sh ; module load etcd && etcdctl endpoint health --cluster --write-out=table +-------------------------+--------+-------------+-------+ | ENDPOINT | HEALTH | TOOK | ERROR | +-------------------------+--------+-------------+-------+ | https://10.141.0.2:2379 | true | 14.480882ms | | +-------------------------+--------+-------------+-------+
Next, for BCM 10+, we have to make sure our Kubernetes API servers are updated again, according to Section 6.
# cm-kubeadm-manage --kube-cluster=default update_configmap # cm-kubeadm-manage --kube-cluster=default update_apiserver headnode # cm-kubeadm-manage --kube-cluster=default update_apiserver node001
The update_apiserver command will ensure that the node will have the appropriate certificates w/r/t communicating with Etcd. (Since these may not be present since we are restoring a backup.)