Installing Kubernetes on Air-Gapped Systems

Kubernetes is most easily installed on a cluster that is able to access the internet. For clusters without internet access it is still possible to deploy Kubernetes, with a few additional steps.

The following document refers to Bright 9.2 on RHEL8.

1. Install the following RPM packages, and their dependencies, on the head node(s) and in all the software images from which Kubernetes is to be deployed:

Packages:

• cm-containerd
• cm-docker
• cm-etcd
• cm-kubernetes-ceph-storage-class (optional)
• cm-kubernetes-local-path-provisioner
• cm-kubernetes-permissions-manager
• cm-kubernetes121
• conntrack-tools
• nginx
• nginx-all-modules
• cm-docker-registry

Dependencies:

• container-selinux
• libnetfilter_cthelper
• libnetfilter_cttimeout
• libnetfilter_queue
• nginx-mod-http-image-filter
• nginx-mod-http-perl
• nginx-mod-http-xslt-filter
• nginx-mod-mail
• nginx-mod-stream
• socat

2. Setup a container registry on the head node using cm-container-registry-setup script:

# cm-container-registry-setup

Note: for simplicity, you can choose docker-registry and deploy it on the head node and choose all default settings.

3. Start docker service to be able to save images to the registry

# systemctl start docker.service

4. Create a list of container images to be loaded into the docker registry

# sed -n ‘s/[ -]*image: *//p’ /cm/local/apps/kubernetes/var/addons/*yaml  | sort -u | tee /tmp/ctr-images.txt docker.io/calico/cni:v3.20.0 docker.io/calico/kube-controllers:v3.20.0 docker.io/calico/node:v3.20.0 docker.io/calico/pod2daemon-flexvol:v3.20.0 docker.io/calico/typha:v3.20.0 docker.io/coredns/coredns:1.8.4 docker.io/jettech/kube-webhook-certgen:v1.5.1 docker.io/kubernetesui/dashboard:v2.3.1 docker.io/kubernetesui/metrics-scraper:v1.0.6 k8s.gcr.io/ingress-nginx/controller:v0.46.0@sha256:52f0058bed0a17ab0fb35628ba97e8d52b5d32299fbc03cc0f6c7b9ff036b61a k8s.gcr.io/kube-state-metrics/kube-state-metrics:v2.1.0 k8s.gcr.io/metrics-server/metrics-server:v0.5.0 nvcr.io/nvidia/k8s-device-plugin:v0.9.0 quay.io/coreos/flannel:v0.14.0 # echo k8s.gcr.io/pause:3.5 >> /tmp/ctr-images.txt

5. Copy the ctr-images.txt to a computer with internet connectivity and running docker.
6. Download and save the list of container images in ctr-images.txt from the computer with internet connectivity:

# mkdir k8s-images && cd k8s-images && for image in $(cat /tmp/ctr-images.txt); do docker pull $image; docker save $image -o ${image//\//_}.tar; done

7. Copy the saved docker images, in tar archive format, into the head node:

8. On the Head Node, import and tag the images into the local Docker registry (installed in step 3)

# module load docker # systemctl start docker # for archive in *.tar; do docker load -i $archive; done # Export the registry_address variable – (replace <$hostname> with the head node hostname) # export registry_address=$(hostname -f):5000; for image in $(cat /tmp/ctr-images.txt); do echo $image | sed “s/docker.io/$registry_address/g;s/k8s.gcr.io/$registry_address/g; s/gcr.io/$registry_address/g;s/quay.io/$registry_address/g;” | xargs echo docker tag $image; done # export registry_address=$(hostname -f):5000; for image in $(cat /tmp/ctr-images.txt); do echo $image | sed “s/docker.io/$registry_address/g;s/k8s.gcr.io/$registry_address/g; s/gcr.io/$registry_address/g;s/quay.io/$registry_address/g;” | xargs docker push; done

9. Run the cm-kubernetes-setup with “–skip-packages” and follow the on-screen instructions to install kubernetes as required but at the end of the wizard choose Save and Exit (not save & deploy).

# cm-kubernetes-setup –skip-packages

10. Replace the official registries with the local one in the generated kubernetes setup config file (default is cm-kubernetes-setup.conf):

# registry_address=$(hostname -f):5000; sed -i “s/docker.io/$registry_address/g;s/k8s.gcr.io/$registry_address/g;s/gcr.io/$registry_address/g;s/quay.io/$registry_address/g;” cm-kubernetes-setup.conf

11. Replace docker registry addresses in the addon yamls

# cp -pr /cm/local/apps/kubernetes/var/addons /cm/local/apps/kubernetes/var/addons.orig # find /cm/local/apps/kubernetes/var/addons/ -type f -name ‘*.yaml’ | xargs -n 1 sed -i “s/docker.io/$registry_address/g;s/k8s.gcr.io/$registry_address/g;s/gcr.io/$registry_address/g;s/quay.io/$registry_address/g;”

12. Install kubernetes using the modified Run cm-kubernetes-setup using the -c option, and wait for the installation to complete:

# cm-kubernetes-setup -c cm-kubernetes-setup.conf

Note: If all is well, Kubernetes gets deployed without issues.

13. Verify that the kubernetes setup is running properly:

# module load kubernetes # kubectl get pod -A NAMESPACE              NAME                                         READY   STATUS      RESTARTS   AGE cmkpm-system           cmkpm-controller-manager-596f8955bf-529ps    2/2     Running     2          2d21h ingress-nginx          ingress-nginx-admission-create-c8nhn         0/1     Completed   0          2d21h ingress-nginx          ingress-nginx-admission-patch-n5nft          0/1     Completed   0          2d21h ingress-nginx          ingress-nginx-controller-67dfb646cd-zxrvz    1/1     Running     0          2d21h kube-system            calico-kube-controllers-58497c65d5-g47m5     1/1     Running     0          2d21h kube-system            calico-node-4h2n7                            1/1     Running     0          2d21h kube-system            calico-node-5f7dx                            1/1     Running     0          2d21h kube-system            calico-node-cvkc2                            1/1     Running     0          2d21h kube-system            calico-node-m6cv4                            1/1     Running     0          2d21h kube-system            coredns-6768db756-j67dk                      1/1     Running     0          2d21h kube-system            coredns-6768db756-qrqsp                      1/1     Running     0          2d21h kube-system            kube-state-metrics-758ccc75d6-8fr48          1/1     Running     0          2d21h kube-system            metrics-server-7b477dd7b9-5mdjn              1/1     Running     0          2d21h kube-system            metrics-server-7b477dd7b9-kbrs8              1/1     Running     0          2d21h kubernetes-dashboard   dashboard-metrics-scraper-69dc9fcfcf-t8ll8   1/1     Running     0          2d21h kubernetes-dashboard   kubernetes-dashboard-6f465b87f7-bbc88        1/1     Running     0          2d21h local-path-storage     local-path-provisioner-544846c679-sw6w8      1/1     Running     0          2d21h [root@adel-b92-rhel8u5-06-28-1 ~]#