Kubernetes is most easily installed on a cluster that is able to access the internet. For clusters without internet access it is still possible to deploy Kubernetes, with a few additional steps.
The following document refers to Bright 9.2 on RHEL8.
- Install the following RPM packages, and their dependencies, on the head node(s) and in all the software images from which Kubernetes is to be deployed:
Packages:
- cm-containerd
- cm-docker
- cm-etcd
- cm-kubernetes-ceph-storage-class (optional)
- cm-kubernetes-local-path-provisioner
- cm-kubernetes-permissions-manager
- cm-kubernetes121
- conntrack-tools
- nginx
- nginx-all-modules
- cm-docker-registry
Dependencies:
- container-selinux
- libnetfilter_cthelper
- libnetfilter_cttimeout
- libnetfilter_queue
- nginx-mod-http-image-filter
- nginx-mod-http-perl
- nginx-mod-http-xslt-filter
- nginx-mod-mail
- nginx-mod-stream
- socat
- Setup a container registry on the head node using cm-container-registry-setup script:
# cm-container-registry-setup |
Note: for simplicity, you can choose docker-registry and deploy it on the head node and choose all default settings.
- Start docker service to be able to save images to the registry
# systemctl start docker.service |
- Create a list of container images to be loaded into the docker registry
# sed -n ‘s/[ -]*image: *//p’ /cm/local/apps/kubernetes/var/addons/*yaml | sort -u | tee /tmp/ctr-images.txt docker.io/calico/cni:v3.20.0 docker.io/calico/kube-controllers:v3.20.0 docker.io/calico/node:v3.20.0 docker.io/calico/pod2daemon-flexvol:v3.20.0 docker.io/calico/typha:v3.20.0 docker.io/coredns/coredns:1.8.4 docker.io/jettech/kube-webhook-certgen:v1.5.1 docker.io/kubernetesui/dashboard:v2.3.1 docker.io/kubernetesui/metrics-scraper:v1.0.6 k8s.gcr.io/ingress-nginx/controller:v0.46.0@sha256:52f0058bed0a17ab0fb35628ba97e8d52b5d32299fbc03cc0f6c7b9ff036b61a k8s.gcr.io/kube-state-metrics/kube-state-metrics:v2.1.0 k8s.gcr.io/metrics-server/metrics-server:v0.5.0 nvcr.io/nvidia/k8s-device-plugin:v0.9.0 quay.io/coreos/flannel:v0.14.0 # echo k8s.gcr.io/pause:3.5 >> /tmp/ctr-images.txt |
- Copy the ctr-images.txt to a computer with internet connectivity and running docker.
- Download and save the list of container images in ctr-images.txt from the computer with internet connectivity:
# mkdir k8s-images && cd k8s-images && for image in $(cat /tmp/ctr-images.txt); do docker pull $image; docker save $image -o ${image//\//_}.tar; done |
- Copy the saved docker images, in tar archive format, into the head node:
- On the Head Node, import and tag the images into the local Docker registry (installed in step 3)
# module load docker # systemctl start docker # for archive in *.tar; do docker load -i $archive; done # Export the registry_address variable – (replace <$hostname> with the head node hostname) # export registry_address=$(hostname -f):5000; for image in $(cat /tmp/ctr-images.txt); do echo $image | sed “s/docker.io/$registry_address/g;s/k8s.gcr.io/$registry_address/g; s/gcr.io/$registry_address/g;s/quay.io/$registry_address/g;” | xargs echo docker tag $image; done # export registry_address=$(hostname -f):5000; for image in $(cat /tmp/ctr-images.txt); do echo $image | sed “s/docker.io/$registry_address/g;s/k8s.gcr.io/$registry_address/g; s/gcr.io/$registry_address/g;s/quay.io/$registry_address/g;” | xargs docker push; done |
- Run the cm-kubernetes-setup with “–skip-packages” and follow the on-screen instructions to install kubernetes as required but at the end of the wizard choose Save and Exit (not save & deploy).
# cm-kubernetes-setup –skip-packages |
- Replace the official registries with the local one in the generated kubernetes setup config file (default is cm-kubernetes-setup.conf):
# registry_address=$(hostname -f):5000; sed -i “s/docker.io/$registry_address/g;s/k8s.gcr.io/$registry_address/g;s/gcr.io/$registry_address/g;s/quay.io/$registry_address/g;” cm-kubernetes-setup.conf |
- Replace docker registry addresses in the addon yamls
# cp -pr /cm/local/apps/kubernetes/var/addons /cm/local/apps/kubernetes/var/addons.orig # find /cm/local/apps/kubernetes/var/addons/ -type f -name ‘*.yaml’ | xargs -n 1 sed -i “s/docker.io/$registry_address/g;s/k8s.gcr.io/$registry_address/g;s/gcr.io/$registry_address/g;s/quay.io/$registry_address/g;” |
- Install kubernetes using the modified Run cm-kubernetes-setup using the -c option, and wait for the installation to complete:
# cm-kubernetes-setup -c cm-kubernetes-setup.conf |
Note: If all is well, Kubernetes gets deployed without issues.
- Verify that the kubernetes setup is running properly:
# module load kubernetes # kubectl get pod -A NAMESPACE NAME READY STATUS RESTARTS AGE cmkpm-system cmkpm-controller-manager-596f8955bf-529ps 2/2 Running 2 2d21h ingress-nginx ingress-nginx-admission-create-c8nhn 0/1 Completed 0 2d21h ingress-nginx ingress-nginx-admission-patch-n5nft 0/1 Completed 0 2d21h ingress-nginx ingress-nginx-controller-67dfb646cd-zxrvz 1/1 Running 0 2d21h kube-system calico-kube-controllers-58497c65d5-g47m5 1/1 Running 0 2d21h kube-system calico-node-4h2n7 1/1 Running 0 2d21h kube-system calico-node-5f7dx 1/1 Running 0 2d21h kube-system calico-node-cvkc2 1/1 Running 0 2d21h kube-system calico-node-m6cv4 1/1 Running 0 2d21h kube-system coredns-6768db756-j67dk 1/1 Running 0 2d21h kube-system coredns-6768db756-qrqsp 1/1 Running 0 2d21h kube-system kube-state-metrics-758ccc75d6-8fr48 1/1 Running 0 2d21h kube-system metrics-server-7b477dd7b9-5mdjn 1/1 Running 0 2d21h kube-system metrics-server-7b477dd7b9-kbrs8 1/1 Running 0 2d21h kubernetes-dashboard dashboard-metrics-scraper-69dc9fcfcf-t8ll8 1/1 Running 0 2d21h kubernetes-dashboard kubernetes-dashboard-6f465b87f7-bbc88 1/1 Running 0 2d21h local-path-storage local-path-provisioner-544846c679-sw6w8 1/1 Running 0 2d21h [root@adel-b92-rhel8u5-06-28-1 ~]# |