Kubernetes is most easily installed on a cluster that is able to access the internet. For clusters without internet access it is still possible to deploy Kubernetes, with a few additional steps.
The following document refers to Bright 9.1.
Install rpm/deb Packages (optional)
Ideally a local mirror of the deb/rpm repository is configured. If that is the case, then you can skip to the next section. Otherwise the following packages (including their dependencies) should be downloaded and installed, both on the head node(s), and in all the software images from which Kubernetes is to be deployed:
- cm-docker
- cm-kubernetes
- cm-etcd
- nginx
- conntrack-tools (on RHEL and Suse) or conntrack (on Ubuntu)
- cm-nvidia-docker (optional)
Take special note that in an HA setup, these packages will need to be installed on BOTH head nodes prior to proceeding.
On RHEL systems, you can use yumdownloader for getting those packages and related dependencies.
Get List of Container Images
From the head node you can create a file with all the images:
sed -n 's/[ -]*image: *//p' /cm/local/apps/kubernetes/var/addons/*yaml | sort -u > images.txt
The file, images.txt, now has all the images:
# cat images.txt
docker.io/calico/cni:v3.10.0
docker.io/calico/kube-controllers:v3.10.0
docker.io/calico/node:v3.10.0
docker.io/calico/pod2daemon-flexvol:v3.10.0
docker.io/calico/typha:v3.10.0
docker.io/coredns/coredns:1.7.0
docker.io/kubernetesui/dashboard:v2.0.0-beta5
docker.io/kubernetesui/metrics-scraper:v1.0.4
docker.io/nvidia/k8s-device-plugin:1.0.0-beta6
k8s.gcr.io/metrics-server/metrics-server:v0.3.7
quay.io/coreos/flannel:v0.12.0-amd64
quay.io/coreos/flannel:v0.12.0-arm
quay.io/coreos/flannel:v0.12.0-arm64
quay.io/coreos/flannel:v0.12.0-ppc64le
quay.io/coreos/flannel:v0.12.0-s390x
quay.io/coreos/kube-state-metrics:v1.9.8
quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.26.1
(Possibly with some duplicates, but that is fine.)
Append the pause image to the file:
echo 'k8s.gcr.io/pause:3.2' >> images.txt
Now the images.txt file must be copied over to a computer with internet connectivity.
Download the Images
From the computer with internet connectivity (and Docker installed), pull all the images:
for image in $(cat images.txt); do docker pull $image; done
Save them in tar archives (this might take some time):
for image in $(cat images.txt); do docker save $image -o ${image//\//_}.tar; done
At this point you’ll have a list of tar archives:
ls *.tar
docker.io_calico_cni:v3.10.0.tar
docker.io_calico_kube-controllers:v3.10.0.tar
docker.io_calico_node:v3.10.0.tar
docker.io_calico_pod2daemon-flexvol:v3.10.0.tar
docker.io_calico_typha:v3.10.0.tar
docker.io_coredns_coredns:1.6.2.tar
docker.io_nvidia_k8s-device-plugin:1.0.0-beta4.tar
k8s.gcr.io_metrics-server-amd64:v0.3.3.tar
k8s.gcr.io_pause:3.1.tar
kubernetesui_dashboard:v2.0.0-beta5.tar
kubernetesui_metrics-scraper:v1.0.1.tar
quay.io_coreos_flannel:v0.11.0-amd64.tar
quay.io_coreos_flannel:v0.11.0-arm64.tar
quay.io_coreos_flannel:v0.11.0-arm.tar
quay.io_coreos_flannel:v0.11.0-ppc64le.tar
quay.io_coreos_flannel:v0.11.0-s390x.tar
quay.io_kubernetes-ingress-controller_nginx-ingress-controller:0.26.1.tar
Those tar archives have to be copied into the head node of the Bright cluster.
On the Head Node
Deploy a local Docker registry
Run cm-docker-registry-setup, and choose the ‘docker-registry.’
cm-docker-registry-setup
Note: in this article we’ll assume the local registry address is “<registry>:5000”.
Provide the images to the local registry
From the head node, load the archives:
module load docker
systemctl start docker
for archive in *.tar; do docker load -i $archive; done
Tag the images:
Export the registry_address variable – (replace <$hostname> with the head node hostname)
export registry_address=<$hostname>:5000
for image in $(cat images.txt); do echo $image | sed "s/docker.io/$registry_address/g;s/k8s.gcr.io/$registry_address/g;s/gcr.io/$registry_address/g;s/quay.io/$registry_address/g;" | xargs docker tag $image; done
Push the images to the local registry:
for image in $(cat images.txt); do echo $image | sed "s/docker.io/$registry_address/g;s/k8s.gcr.io/$registry_address/g;s/gcr.io/$registry_address/g;s/quay.io/$registry_address/g;" | xargs docker push; done
Run the Kubernetes Setup
Run the cm-kubernetes-setup wizard to create the required configuration file. But don’t choose to “Save & Deploy”. Instead, choose to “Save config & Exit”.
cm-kubernetes-setup
Replace the official registries with the local one:
sed -i "s/docker.io/$registry_address/g;s/k8s.gcr.io/$registry_address/g;s/gcr.io/$registry_address/g;s/quay.io/$registry_address/g;" cm-kubernetes-setup.conf
Backup addons yaml directory:
cp -pr /cm/local/apps/kubernetes/var/addons /cm/local/apps/kubernetes/var/addons.bak
Replace docker registry addresses in the addon yamls as well:
find /cm/local/apps/kubernetes/var/addons/ -type f -name '*.yaml' | xargs -n 1 sed -i "s/docker.io/$registry_address/g;s/k8s.gcr.io/$registry_address/g;s/gcr.io/$registry_address/g;s/quay.io/$registry_address/g;"
To double check that the replace went well you can do a quick grep and see if everything points to our docker registry:
grep image /cm/local/apps/kubernetes/var/addons
Also, in the same cm-kubernetes-setup.conf file you should add the last line to the Kubelet section (again, replacing <$hostname> with the value of your head node hostname:
node:
kubelet_port: 10250
options:
- --volume-stats-agg-period=0
- --pod-infra-container-image=<$hostname>:5000/pause:3.2
If the packages have been already installed, then let us change this key/value setting in the same file:
skip_packages: true
(Note that it has to be changed in two places in the config file)
Run cm-kubernetes-setup with the -c option, and wait for the installation to complete:
cm-kubernetes-setup -c cm-kubernetes-setup.conf
If all is well, Kubernetes gets deployed without issues.
Finally you want to restore the backup dir:
mv /cm/local/apps/kubernetes/var/addons /cm/local/apps/kubernetes/var/addons.airgapped
mv /cm/local/apps/kubernetes/var/addons.bak /cm/local/apps/kubernetes/var/addons
Example result:
[root@rb-airgapped ~]# module load kubernetes
[root@rb-airgapped ~]# kubectl get pod -n kube-system
NAME READY STATUS RESTARTS AGE
calico-kube-controllers-75dcf56fc6-gg98j 1/1 Running 0 47s
calico-node-4bz7k 1/1 Running 0 47s
calico-node-4czdc 1/1 Running 0 47s
calico-node-6vwmd 1/1 Running 0 47s
calico-node-d6f66 1/1 Running 0 47s
calico-node-dmx4k 1/1 Running 0 47s
calico-node-qnh5h 1/1 Running 0 47s
calico-node-rtntq 1/1 Running 0 47s
coredns-785b8b6779-5hxfr 1/1 Running 0 47s
coredns-785b8b6779-j8ppm 1/1 Running 0 47s
metrics-server-6db957d6f7-g9xvf 1/1 Running 0 41s
metrics-server-6db957d6f7-zddt4 1/1 Running 0 41s