Overview
Base Command Manager (BCM) supports Cluster API for deploying Kubernetes clusters. This feature was introduced in BCM 10 and is documented in containerization manual chapter 8.
Problem: Upstream changes in the Cluster API project and discontinued maintenance of the CAPI infrastructure provider (Bring Your Own Host) broke Cluster API compatibility after Kubernetes version 1.30. Clusters created with newer Kubernetes versions get stuck during provisioning:
[root@headnode ~]# kubectl get machine NAME CLUSTER NODENAME PROVIDERID PHASE AGE VERSION mycluster1-control-plane-9ldks mycluster1 Provisioning ... v1.31.1 mycluster1-md-0-cl6tl-pkgxq mycluster1 Pending ... v1.31.1
Root Causes:
- Two installation script secrets conflict with newer Kubernetes versions and need disambiguation
- Kube VIP encounters issues due to administrator RBAC changes after super-admin was introduced in v1.31
Solution: We now maintain an internal fork to keep components up-to-date and compatible with the latest Cluster API changes from the upstream Kubernetes SIG. The next BCM 10 release will include this updated Cluster API version, enabling support for recent Kubernetes versions.
This KB article provides instructions for updating CAPI components on existing clusters and explains the slight changes to the containerization manual instructions.
Prerequisites
- Supported Distributions: RHEL- and Ubuntu-based distributions
- Required BCM Version: 10.25.03
- Action Required: Update to BCM 10.25.03 before proceeding
Verify Your BCM Version
Confirm you’re running BCM 10.25.03:
root@headnode# cm-package-release-info -f cm-setup
Name Version Release(s)
-------- --------- ------------
cm-setup 120796 10.25.03
If not at version 10.25.03, update BCM before continuing.
Package Installation
Step 1: Download Required Packages
Download the following two packages to /tmp on your head node:
- cm-capi: Contains updated clusterctl and related files
- cm-setup: Contains the updated
cm-kubernetes-capi-setupwizard
Choose the appropriate download commands for your distribution:
RHEL 8 / Rocky 8
wget -O /tmp/cm-capi-1.10.2-100017_cm10.0_d2f1da6223.x86_64.rpm https://support2.brightcomputing.com/capi/rhel8/cm-capi-1.10.2-100017_cm10.0_d2f1da6223.x86_64.rpm
wget -O /tmp/cm-setup-10.0-120799_cm10.0_049019c9f2.x86_64.rpm https://support2.brightcomputing.com/capi/rhel8/cm-setup-10.0-120799_cm10.0_049019c9f2.x86_64.rpm
RHEL 9 / Rocky 9
wget -O /tmp/cm-capi-1.10.2-100017_cm10.0_d2f1da6223.x86_64.rpm https://support2.brightcomputing.com/capi/rhel9/cm-capi-1.10.2-100017_cm10.0_d2f1da6223.x86_64.rpm
wget -O /tmp/cm-setup-10.0-120799_cm10.0_049019c9f2.x86_64.rpm https://support2.brightcomputing.com/capi/rhel9/cm-setup-10.0-120799_cm10.0_049019c9f2.x86_64.rpm
SLES 15
wget -O /tmp/cm-capi-1.10.2-100017_cm10.0_d2f1da6223.x86_64.rpm https://support2.brightcomputing.com/capi/sles15/cm-capi-1.10.2-100017_cm10.0_d2f1da6223.x86_64.rpm
wget -O /tmp/cm-setup-10.0-120799_cm10.0_049019c9f2.x86_64.rpm https://support2.brightcomputing.com/capi/sles15/cm-setup-10.0-120799_cm10.0_049019c9f2.x86_64.rpm
Ubuntu 20.04
wget -O /tmp/cm-capi_1.10.2-100017-cm10.0-d2f1da6223_amd64.deb https://support2.brightcomputing.com/capi/ubuntu2004/cm-capi_1.10.2-100017-cm10.0-d2f1da6223_amd64.deb
wget -O /tmp/cm-setup_10.0-120799-cm10.0-049019c9f2_amd64.deb https://support2.brightcomputing.com/capi/ubuntu2004/cm-setup_10.0-120799-cm10.0-049019c9f2_amd64.deb
Ubuntu 22.04
wget -O /tmp/cm-capi_1.10.2-100017-cm10.0-d2f1da6223_amd64.deb https://support2.brightcomputing.com/capi/ubuntu2204/cm-capi_1.10.2-100017-cm10.0-d2f1da6223_amd64.deb
wget -O /tmp/cm-setup_10.0-120799-cm10.0-049019c9f2_amd64.deb https://support2.brightcomputing.com/capi/ubuntu2204/cm-setup_10.0-120799-cm10.0-049019c9f2_amd64.deb
Ubuntu 24.04
wget -O /tmp/cm-capi_1.10.2-100017-cm10.0-d2f1da6223_amd64.deb https://support2.brightcomputing.com/capi/ubuntu2404/cm-capi_1.10.2-100017-cm10.0-d2f1da6223_amd64.deb
wget -O /tmp/cm-setup_10.0-120799-cm10.0-049019c9f2_amd64.deb https://support2.brightcomputing.com/capi/ubuntu2404/cm-setup_10.0-120799-cm10.0-049019c9f2_amd64.deb
Step 2: Verify Package Integrity
Verify the downloaded package checksums (below is an example for RHEL8):
md5sum /tmp/cm-capi-1.10.2-100017_cm10.0_d2f1da6223.x86_64.rpm /tmp/cm-setup-10.0-120799_cm10.0_049019c9f2.x86_64.rpm
Compare the output with these expected checksums:
| Distro | Checksum | Package |
|---|---|---|
| rhel8 | 842c1c11fbd870921cda0ec67dbdaeb8 | cm-capi-1.10.2-100017_cm10.0_d2f1da6223.x86_64.rpm |
| rhel8 | 5bf5fa3a9eea99a5a9c2730a61c50544 | cm-setup-10.0-120799_cm10.0_049019c9f2.x86_64.rpm |
| rhel9 | 223336984ca0a43054296fd40c1a1cda | cm-capi-1.10.2-100017_cm10.0_d2f1da6223.x86_64.rpm |
| rhel9 | 0ca16a7fac101767530b6d285c3af1f7 | cm-setup-10.0-120799_cm10.0_049019c9f2.x86_64.rpm |
| sles15 | cdd12e01b6b574c3e51a29b549f5b653 | cm-capi-1.10.2-100017_cm10.0_d2f1da6223.x86_64.rpm |
| sles15 | 8c44eadfab92039aa125c623ca180676 | cm-setup-10.0-120799_cm10.0_049019c9f2.x86_64.rpm |
| ubuntu2004 | 6b4eaebbc794fa739f4f6598159f439e | cm-capi_1.10.2-100017-cm10.0-d2f1da6223_amd64.deb |
| ubuntu2004 | 124e8e8d9c3bad277e8e5653d75d141e | cm-setup_10.0-120799-cm10.0-049019c9f2_amd64.deb |
| ubuntu2204 | 54a755a0a6020c1a7a5e5893d1e236b3 | cm-capi_1.10.2-100017-cm10.0-d2f1da6223_amd64.deb |
| ubuntu2204 | 31a343e43399520404891cf1ae64ef65 | cm-setup_10.0-120799-cm10.0-049019c9f2_amd64.deb |
| ubuntu2404 | e5b33d193df02061e62635d9fd211053 | cm-capi_1.10.2-100017-cm10.0-d2f1da6223_amd64.deb |
| ubuntu2404 | d9077473c87fd99048c7884cdf4dc7d1 | cm-setup_10.0-120799-cm10.0-049019c9f2_amd64.deb |
Step 3: Install Packages
Ubuntu Systems
Install the updated cm-capi package:
root@headnode:/tmp# apt install ./cm-capi_1.10.2-100017-cm10.0-d2f1da6223_amd64.deb
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
Note, selecting 'cm-capi' instead of './cm-capi_1.10.2-100017-cm10.0-d2f1da6223_amd64.deb'
The following packages will be upgraded:
cm-capi
1 upgraded, 0 newly installed, 0 to remove and 274 not upgraded.
Need to get 0 B/51.0 MB of archives.
After this operation, 25.1 MB of additional disk space will be used.
Get:1 /tmp/cm-capi_1.10.2-100017-cm10.0-d2f1da6223_amd64.deb cm-capi amd64 1.10.2-100017-cm10.0-d2f1da6223 [51.0 MB]
(Reading database ... 255618 files and directories currently installed.)
Preparing to unpack .../cm-capi_1.10.2-100017-cm10.0-d2f1da6223_amd64.deb ...
Unpacking cm-capi (1.10.2-100017-cm10.0-d2f1da6223) over (1.4.3-100015-cm10.0-d709a7aa98) ...
Setting up cm-capi (1.10.2-100017-cm10.0-d2f1da6223) ...
Install the updated cm-setup package:
root@headnode:/tmp# apt install ./cm-setup_10.0-120799-cm10.0-049019c9f2_amd64.deb
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
Note, selecting 'cm-setup' instead of './cm-setup_10.0-120799-cm10.0-049019c9f2_amd64.deb'
The following packages will be upgraded:
cm-setup
1 upgraded, 0 newly installed, 0 to remove and 274 not upgraded.
Need to get 0 B/16.5 MB of archives.
After this operation, 3,072 B disk space will be freed.
Get:1 /tmp/cm-setup_10.0-120799-cm10.0-049019c9f2_amd64.deb cm-setup amd64 10.0-120799-cm10.0-049019c9f2 [16.5 MB]
(Reading database ... 255620 files and directories currently installed.)
Preparing to unpack .../cm-setup_10.0-120799-cm10.0-049019c9f2_amd64.deb ...
Unpacking cm-setup (10.0-120799-cm10.0-049019c9f2) over (10.0-120796-cm10.0-d4d2ffc8ea) ...
Setting up cm-setup (10.0-120799-cm10.0-049019c9f2) ...
RHEL/Rocky Systems
Use yum instead of apt with the same package files.
SLES Systems
Use zypper instead of apt with the same package files.
CAPI Management Cluster Operations
New Management Cluster Setup
To configure a new CAPI management cluster, follow the standard instructions from the containerization manual. The updated components will automatically support newer Kubernetes versions.
Existing Management Cluster Upgrade
Follow these steps to upgrade an existing management cluster:
Configure Kubernetes Context
- Load the Kubernetes module for your management cluster:
root@headnode:~# module load kubernetes/mgmt/...
- Verify kubectl connectivity and check existing components:
kubectl get pod -A | grep -E '^(byoh-system|capi-|cert-manager)'
- Load the CAPI module:
root@headnode:~# module load capi
- Verify the clusterctl version (should be v1.10.2):
root@headnode:~# clusterctl version clusterctl version: &version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.2", GitCommit:"525566440a77696f6d57535229987ad226a5e51a", GitTreeState:"clean", BuildDate:"2025-05-20T16:44:07Z", GoVersion:"go1.23.8", Compiler:"gc", Platform:"linux/amd64"}
Perform CAPI Upgrade
- Execute the cluster upgrade:
root@headnode:~# clusterctl upgrade apply --core cluster-api:v1.10.2 --bootstrap kubeadm:v1.10.2 --control-plane kubeadm:v1.10.2 --infrastructure byoh:v0.5.0 Checking if cert-manager needs upgrade... Deleting cert-manager version="v1.12.1" Installing cert-manager version="v1.17.2" Waiting for cert-manager to be available... Performing upgrade... Scaling down Provider={"name":"cluster-api","namespace":"capi-system"} providerVersion="" Scaling down Provider={"name":"bootstrap-kubeadm","namespace":"capi-kubeadm-bootstrap-system"} providerVersion="" Scaling down Provider={"name":"control-plane-kubeadm","namespace":"capi-kubeadm-control-plane-system"} providerVersion="" Scaling down Provider={"name":"infrastructure-byoh","namespace":"byoh-system"} providerVersion="" Deleting Provider={"name":"cluster-api","namespace":"capi-system"} providerVersion="" Installing provider="cluster-api" version="v1.10.2" targetNamespace="capi-system" Deleting Provider={"name":"bootstrap-kubeadm","namespace":"capi-kubeadm-bootstrap-system"} providerVersion="" Installing provider="bootstrap-kubeadm" version="v1.10.2" targetNamespace="capi-kubeadm-bootstrap-system" Deleting Provider={"name":"control-plane-kubeadm","namespace":"capi-kubeadm-control-plane-system"} providerVersion="" Installing provider="control-plane-kubeadm" version="v1.10.2" targetNamespace="capi-kubeadm-control-plane-system" Deleting Provider={"name":"infrastructure-byoh","namespace":"byoh-system"} providerVersion="" Installing provider="infrastructure-byoh" version="v0.5.0" targetNamespace="byoh-system"
- Apply infrastructure components configuration:
root@headnode:~# kubectl apply -f /cm/local/apps/capi/var/infrastructure-components.yaml
- Update the byoh-controller-manager container image:
kubectl set image deployment/byoh-controller-manager manager=brightcomputing/cluster-api-byoh-controller:v3.0.0 -n byoh-system
Verify Upgrade Success
Confirm all components are running properly:
root@headnode:~# kubectl get pod -A | grep -E '^(byoh-system|capi-|cert-manager)'
byoh-system byoh-controller-manager-7f54957cc9-q2qwv 2/2 Running 0 94s
capi-kubeadm-bootstrap-system capi-kubeadm-bootstrap-controller-manager-5767c46798-f8g2z 1/1 Running 0 93s
capi-kubeadm-control-plane-system capi-kubeadm-control-plane-controller-manager-7cd986946d-r6shk 1/1 Running 0 93s
capi-system capi-controller-manager-c4c7c7bf7-msv9d 1/1 Running 0 92s
cert-manager cert-manager-776869779d-77vgf 1/1 Running 0 4h55m
cert-manager cert-manager-cainjector-b6fb648df-76v8w 1/1 Running 0 4h55m
cert-manager cert-manager-webhook-5d95f5d4c5-fnbq7 1/1 Running 0 4h55m
All pods should show Running status. The upgrade is now complete.
Workload Cluster Creation
Updated Command Syntax
The process for creating new workload clusters has changed slightly. You must now specify an explicit template file.
Previous Method (No Longer Correct)
module load capi
CONTROL_PLANE_ENDPOINT_IP=10.141.168.1 clusterctl generate cluster mycluster1 \
--infrastructure byoh \
--kubernetes-version v1.31.1 \
--control-plane-machine-count 1 \
--worker-machine-count 1 > cluster.yaml
New Required Method
module load capi
CONTROL_PLANE_ENDPOINT_IP=10.141.168.1 clusterctl generate cluster mycluster1 \
--from /cm/local/apps/capi/var/cluster-template.yaml \
--kubernetes-version v1.31.1 \
--control-plane-machine-count 1 \
--worker-machine-count 1 > cluster.yaml
Key Change: The --from flag is now required to specify the cluster template location.
Deploy the Workload Cluster
After generating the cluster configuration, inspect and deploy it:
root@headnode~# kubectl apply -f cluster.yaml
The generated cluster.yaml includes an additional static pod manifest to patch kube-vip, which provides the virtual IP for the Kubernetes API server.
Summary
This update resolves Kubernetes compatibility issues beyond version 1.30 by:
- Updating Cluster API components to v1.10.2
- Updating Cluster API infrastructure provider to v3.0.0.
[root@headnode ~]# kubectl get machine NAME CLUSTER NODENAME PROVIDERID PHASE AGE VERSION mycluster1-control-plane-9ldks mycluster1 node002 byoh://node002/c8ooiq Running ... v1.31.1 mycluster1-md-0-cl6tl-pkgxq mycluster1 node004 byoh://node004/oih71v Running ... v1.31.1