This KB article aims to give answers to common questions surrounding this topic.
1. Prerequisites
- This KB article assumes Kubernetes is deployed with Bright Cluster Manager version 9.2 or below.
2. How Bright Cluster Manager manages kubeconfig files for users
More details can also be found our admin manual, but we’ll start with a quick summary. This section is optional and can be skipped.
The Kubernetes deployed by Bright is heavily inspired by the guide Kubernetes the hard way (by Kelsey Hightower) (link here). The guide aims to setup a production ready Kubernetes.
For the root user a “full admin” user is created, and more details about how this certificate is created can be found in the aforementioned guide in chapter four. In short, a certificate is created inside /root/.kube by Bright Cluster Manager on all the nodes that are part of the particular Kubernetes deployment.
The certificate is created, and has the Kubernetes cluster name as part of the file name. The same applies to the kubeconfig file, which is also created by Bright Cluster Manager.
[root@headnode ~]# ls -al /root/.kube
total 20
drwx------ 3 root root 91 Jul 4 12:34 .
dr-xr-x--- 13 root root 4096 Jul 2 06:42 ..
-r--r----- 1 root root 1708 Jul 4 12:34 admin-default.key
-r--r----- 1 root root 1696 Jul 4 12:34 admin-default.pem
-rw------- 1 root root 7672 Jul 4 12:58 config-defaultThe config-default (the kubeconfig file) actually embeds the two certificate files as base64. This makes the single kubeconfig file more portable (easy to copy around), the certificates are there for convenience, but it is important to know that they are not used by the kubeconfig file.
For regular users the certificates are also placed in the home directory, under .kube. For example: /home/cmsupport/.kube. The active Head Node is the only node where these are written by Bright Cluster Manager (this is different from the full admin user that resides in /root, since the home directories are shared across nodes).
[root@headnode ~]# ls -al /home/cmsupport/.kube
total 24
drwx------ 3 cmsupport cmsupport 111 Jul 4 13:38 .
drwxr-xr-x 5 cmsupport cmsupport 306 Jul 4 13:12 ..
-rw------- 1 cmsupport cmsupport 6739 Jul 4 13:21 config-default
-r--r----- 1 cmsupport cmsupport 1704 Jul 4 12:43 default.key
-r--r----- 1 cmsupport cmsupport 1720 Jul 4 12:43 default.pemIn order to use kubectl, as root, or a regular user that was given access, the appropriate module file needs to be loaded. Highlighted below are the PATH to add kubectl to the path, and KUBECONFIG to instruct kubectl which kubeconfig to load.
[cmsupport@headnode ~]$ module show kubernetes/default/1.21.4
------------------------------------------------------
/cm/local/modulefiles/kubernetes/default/1.21.4:
------------------------------------------------------
whatis("Adds kubernetes to your environment ")
conflict("kubernetes")
prepend_path("PATH","/cm/local/apps/kubernetes/current/bin")
prepend_path("MANPATH","/cm/local/apps/kubernetes/current/man")
setenv("CALICO_DATASTORE_TYPE","kubernetes")
setenv("CALICO_KUBECONFIG","/home/cmsupport/.kube/config-default")
setenv("KUBECONFIG","/home/cmsupport/.kube/config-default")
prepend_path("PATH","/cm/local/apps/etcd/current/bin")
prepend_path("MANPATH","/cm/local/apps/etcd/current/man")
setenv("ETCDCTL_API","3")
setenv("ETCDCTL_ENDPOINTS","https://10.141.0.1:2379")
setenv("ETCDCTL_CACERT","/cm/local/apps/kubernetes/var/etc/kubeca-default.pem")
setenv("ETCDCTL_CERT","/cm/local/apps/etcd/var/etc/client-default.pem")
setenv("ETCDCTL_KEY","/cm/local/apps/etcd/var/etc/client-default.key")
help([[ Adds kubernetes to your environment
]])3. Changing the default context in a user kubeconfig
By default, each user has their default context set to their personal restricted namespace.
[cmsupport@headnode ~]$ module load kubernetes/default/1.21.4
[cmsupport@headnode ~]$ kubectl get pod
No resources found in cmsupport-restricted namespace.The kubectl config set-context feature.
We can change the default context to a different namespace (that we have access to), so we don’t have to keep specify the namespace we’re working with for every command. The kubectl utility has a command for that, and it manipulates the active kubeconfig config file for this purpose.
# We will change the namespace for the current context as an example
[cmsupport@headnode ~]$ kubectl config set-context --namespace kube-system --current
Context "cmsupport@restricted" modified.
[cmsupport@headnode ~]$ kubectl get pod
NAME READY STATUS RESTARTS AGE
calico-kube-controllers-58497c65d5-2dpbc 1/1 Running 7 3d21h
calico-node-fmhqn 1/1 Running 1 3d21h
calico-node-j4kzm 1/1 Running 1 3d21h
calico-node-rbk9w 1/1 Running 1 3d21h
coredns-6768db756-8kp7f 1/1 Running 1 3d21h
coredns-6768db756-g9fst 1/1 Running 1 3d21h
kube-state-metrics-758ccc75d6-6khnr 1/1 Running 2 3d21h
metrics-server-7b477dd7b9-jnwtd 1/1 Running 1 3d21h
metrics-server-7b477dd7b9-zm58v 1/1 Running 2 3d21hNow the user doesn’t have to constantly provide --namespace kube-system, or -n kube-system to their commands while working in kube-system namespace.
The problem
Bright Cluster Manager manages the users kubeconfig file, so the change made to /home/cmsupport/.kube/config-default will be overwritten by Bright Cluster Manager after a while, resetting the context back to the cmsupport-restricted namespace.
Solution for Bright Cluster Manager 9.2
In order to avoid this behavior, a flag was introduced in Bright Cluster Manager 9.2, which can be toggled via cmsh as follows.
[root@headnode ~]# cmsh
[headnode]% kubernetes
[headnode->kubernetes[default]]% users
[headnode->kubernetes[default]->user]% list
User name (key) Manage kubeconfig
--------------- -----------------
cmsupport yes
[headnode->kubernetes[default]->user]% set cmsupport managekubeconfig no
[headnode->kubernetes*[default*]->user*]% commit
[headnode->kubernetes[default]->user]% list
User name (key) Manage kubeconfig
--------------- -----------------
cmsupport no
[headnode->kubernetes[default]->user]% After the above commands, Bright Cluster Manager will no longer overwrite the changes made with kubectl config.
Solution for Bright Cluster Manager version <= 9.1
Since the feature mentioned in the above block is not available in 9.1, we can work around it differently. We can change the module file for Kubernetes in cmsh, as follows.
[root@headnode ~]# cmsh
[headnode]% kubernetes
[headnode->kubernetes[default]]% set modulefiletemplate The above will open up an editor that allows for editing of the module file template. We need to replace the following line.
setenv KUBECONFIG "${KUBECONFIG}"With the following block of TCL code.
if { [file exists ${KUBECONFIG}.unmanaged] } {
setenv KUBECONFIG "${KUBECONFIG}.unmanaged"
} else {
setenv KUBECONFIG "${KUBECONFIG}"
}This change is compatible with env-modules, as well as Lmod. Then we save the change, and commit the change in cmsh.
[headnode->kubernetes*[default*]]% commitNothing should change in behavior with regards to the module file. If a user does nothing special, they will get the original Bright managed kubeconfig file as before.
[root@headnode ~]# su - cmsupport
Last login: Mon Jul 4 13:37:38 CEST 2022 on pts/1
[cmsupport@headnode ~]$ module load kubernetes/default/1.21.4
# the kube config file is as ever before
[cmsupport@headnode ~]$ echo $KUBECONFIG
/home/cmsupport/.kube/config-defaultHowever, they can decide to switch to an unmanaged kubeconfig file by copying their kubeconfig file as follows.
# the sysadmin can also do it for the user
[cmsupport@headnode ~]$ cp .kube/config-default .kube/config-default.unmanaged
# unload + load, or use switch, to load the module file again
[cmsupport@headnode ~]$ module switch kubernetes/default/1.21.4
# the unmanaged file should be used instead
[cmsupport@headnode ~]$ echo $KUBECONFIG
/home/cmsupport/.kube/config-default.unmanagedThe module file will use the unmanaged version when present, and the kubectl config command will manipulate the config-default.unmanaged file instead.
Future versions
We will likely make this change part of the Kubernetes module file in the near future, however at the time of writing the module file needs to be changed manually. When future Bright versions support it out of the box, this KB article will be updated.