Installing Kubernetes on Air-Gapped Systems

Contents

Kubernetes is most easily installed on a cluster that is able to access the internet. For clusters without internet access it is still possible to deploy Kubernetes, with a few additional steps.

The following document refers to Bright 9.1.

Install rpm/deb Packages (optional)

Ideally a local mirror of the deb/rpm repository is configured. If that is the case, then you can skip to the next section. Otherwise the following packages (including their dependencies) should be downloaded and installed, both on the head node(s), and in all the software images from which Kubernetes is to be deployed:

cm-docker
cm-kubernetes
cm-etcd
nginx
conntrack-tools (on RHEL and Suse) or conntrack (on Ubuntu)
cm-nvidia-docker (optional)

Take special note that in an HA setup, these packages will need to be installed on BOTH head nodes prior to proceeding.

On RHEL systems, you can use yumdownloader for getting those packages and related dependencies.

Get List of Container Images

From the head node you can create a file with all the images:

sed -n 's/[ -]*image: *//p' /cm/local/apps/kubernetes/var/addons/*yaml | sort -u > images.txt

The file, images.txt, now has all the images:

# cat images.txt
docker.io/calico/cni:v3.10.0
docker.io/calico/kube-controllers:v3.10.0
docker.io/calico/node:v3.10.0
docker.io/calico/pod2daemon-flexvol:v3.10.0
docker.io/calico/typha:v3.10.0
docker.io/coredns/coredns:1.7.0
docker.io/kubernetesui/dashboard:v2.0.0-beta5
docker.io/kubernetesui/metrics-scraper:v1.0.4
docker.io/nvidia/k8s-device-plugin:1.0.0-beta6
k8s.gcr.io/metrics-server/metrics-server:v0.3.7
quay.io/coreos/flannel:v0.12.0-amd64
quay.io/coreos/flannel:v0.12.0-arm
quay.io/coreos/flannel:v0.12.0-arm64
quay.io/coreos/flannel:v0.12.0-ppc64le
quay.io/coreos/flannel:v0.12.0-s390x
quay.io/coreos/kube-state-metrics:v1.9.8
quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.26.1

(Possibly with some duplicates, but that is fine.)

Append the pause image to the file:

echo 'k8s.gcr.io/pause:3.2' >> images.txt

Now the images.txt file must be copied over to a computer with internet connectivity.

Download the Images

From the computer with internet connectivity (and Docker installed), pull all the images:

for image in $(cat images.txt); do docker pull $image; done

Save them in tar archives (this might take some time):

for image in $(cat images.txt); do docker save $image -o ${image//\//_}.tar; done

At this point you’ll have a list of tar archives:

ls *.tar 
docker.io_calico_cni:v3.10.0.tar
docker.io_calico_kube-controllers:v3.10.0.tar
docker.io_calico_node:v3.10.0.tar
docker.io_calico_pod2daemon-flexvol:v3.10.0.tar
docker.io_calico_typha:v3.10.0.tar
docker.io_coredns_coredns:1.6.2.tar
docker.io_nvidia_k8s-device-plugin:1.0.0-beta4.tar
k8s.gcr.io_metrics-server-amd64:v0.3.3.tar
k8s.gcr.io_pause:3.1.tar
kubernetesui_dashboard:v2.0.0-beta5.tar
kubernetesui_metrics-scraper:v1.0.1.tar
quay.io_coreos_flannel:v0.11.0-amd64.tar
quay.io_coreos_flannel:v0.11.0-arm64.tar
quay.io_coreos_flannel:v0.11.0-arm.tar
quay.io_coreos_flannel:v0.11.0-ppc64le.tar
quay.io_coreos_flannel:v0.11.0-s390x.tar
quay.io_kubernetes-ingress-controller_nginx-ingress-controller:0.26.1.tar

Those tar archives have to be copied into the head node of the Bright cluster.

On the Head Node

Deploy a local Docker registry

Run cm-docker-registry-setup, and choose the ‘docker-registry.’

cm-docker-registry-setup

Note: in this article we’ll assume the local registry address is “<registry>:5000”.

Provide the images to the local registry

From the head node, load the archives:

module load docker 
systemctl start docker
for archive in *.tar; do docker load -i $archive; done

Tag the images:

Export the registry_address variable – (replace <$hostname> with the head node hostname)

export registry_address=<$hostname>:5000

for image in $(cat images.txt); do echo $image | sed "s/docker.io/$registry_address/g;s/k8s.gcr.io/$registry_address/g;s/gcr.io/$registry_address/g;s/quay.io/$registry_address/g;" | xargs docker tag $image; done

Push the images to the local registry:

for image in $(cat images.txt); do echo $image | sed "s/docker.io/$registry_address/g;s/k8s.gcr.io/$registry_address/g;s/gcr.io/$registry_address/g;s/quay.io/$registry_address/g;" | xargs docker push; done

Run the Kubernetes Setup

Run the cm-kubernetes-setup wizard to create the required configuration file. But don’t choose to “Save & Deploy”. Instead, choose to “Save config & Exit”.

cm-kubernetes-setup

Replace the official registries with the local one:

sed -i "s/docker.io/$registry_address/g;s/k8s.gcr.io/$registry_address/g;s/gcr.io/$registry_address/g;s/quay.io/$registry_address/g;" cm-kubernetes-setup.conf

Backup addons yaml directory:

cp -pr /cm/local/apps/kubernetes/var/addons /cm/local/apps/kubernetes/var/addons.bak

Replace docker registry addresses in the addon yamls as well:

find /cm/local/apps/kubernetes/var/addons/ -type f -name '*.yaml' | xargs -n 1 sed -i "s/docker.io/$registry_address/g;s/k8s.gcr.io/$registry_address/g;s/gcr.io/$registry_address/g;s/quay.io/$registry_address/g;"

To double check that the replace went well you can do a quick grep and see if everything points to our docker registry:

grep image /cm/local/apps/kubernetes/var/addons

Also, in the same cm-kubernetes-setup.conf file you should add the last line to the Kubelet section (again, replacing <$hostname> with the value of your head node hostname:

   node:
     kubelet_port: 10250
     options:
     - --volume-stats-agg-period=0
     - --pod-infra-container-image=<$hostname>:5000/pause:3.2

If the packages have been already installed, then let us change this key/value setting in the same file:

skip_packages: true

(Note that it has to be changed in two places in the config file)

Run cm-kubernetes-setup with the -c option, and wait for the installation to complete:

cm-kubernetes-setup -c cm-kubernetes-setup.conf

If all is well, Kubernetes gets deployed without issues.

Finally you want to restore the backup dir:

mv /cm/local/apps/kubernetes/var/addons /cm/local/apps/kubernetes/var/addons.airgapped
mv /cm/local/apps/kubernetes/var/addons.bak /cm/local/apps/kubernetes/var/addons

Example result:

[root@rb-airgapped ~]# module load kubernetes
[root@rb-airgapped ~]# kubectl get pod -n kube-system
NAME                                       READY   STATUS    RESTARTS   AGE
calico-kube-controllers-75dcf56fc6-gg98j   1/1     Running   0          47s
calico-node-4bz7k                          1/1     Running   0          47s
calico-node-4czdc                          1/1     Running   0          47s
calico-node-6vwmd                          1/1     Running   0          47s
calico-node-d6f66                          1/1     Running   0          47s
calico-node-dmx4k                          1/1     Running   0          47s
calico-node-qnh5h                          1/1     Running   0          47s
calico-node-rtntq                          1/1     Running   0          47s
coredns-785b8b6779-5hxfr                   1/1     Running   0          47s
coredns-785b8b6779-j8ppm                   1/1     Running   0          47s
metrics-server-6db957d6f7-g9xvf            1/1     Running   0          41s
metrics-server-6db957d6f7-zddt4            1/1     Running   0          41s

Updated on June 2, 2022