1. Prerequisites
- This article was written with 9.2 in mind, but the content should apply for older versions as well.
- Kubernetes is deployed with either
docker
orcontainerd
. - We’re assuming all the Kubernetes nodes do not have external internet access without a configured HTTP(s) proxy.
2. Issues potentially addressed by this KB article
- Kubernetes cannot pull container images (and because of it cannot create containers for Pods)
We will make the container runtime use an http proxy. - The software running inside the containers do not have internet access.
We will expose http proxy environment variables inside the container.
Please note that setting these environment variables does not guarantee that they will actually be used by the software. Although it’s common practice that many tools adhere to these environment variables, not all tools might implement this feature.
3. Figure out the correct environment variables
These are probably known to your organization, and something similar to:
http_proxy=http://myuser:password@1.2.3.4:3128
https_proxy=http://myuser:password@1.2.3.4:3128
no_proxy=10.0.0.0/8,192.168.0.0/16,172,16.0.0/12
4. Configuring the proxy for the containerd
or docker
service
For the sake of simplicity of this KB article we have only one category to deal with. Please repeat these steps for Head Nodes in case they are involved, or any additional categories that may be involved.
First the steps for containerd
. The steps for docker are almost the same.
# chroot into the software image
cm-chroot-sw-img /cm/images/default-image/
# create directory to put an override file for containerd
mkdir -p /etc/systemd/system/containerd.service.d
# create the file
cat << EOF > /etc/systemd/system/containerd.service.d/override.conf
[Service]
Environment=http_proxy=http://myuser:password@1.2.3.4:3128
Environment=https_proxy=http://myuser:password@1.2.3.4:3128
Environment=no_proxy=10.0.0.0/8,192.168.0.0/16,172,16.0.0/12
EOF
# exit the chroot
exit
For docker
use:
cm-chroot-sw-img /cm/images/default-image/
mkdir -p /etc/systemd/system/docker.service.d
cat << EOF > /etc/systemd/system/docker.service.d/override.conf
[Service]
Environment=http_proxy=http://myuser:password@1.2.3.4:3128
Environment=https_proxy=http://myuser:password@1.2.3.4:3128
Environment=no_proxy=10.0.0.0/8,192.168.0.0/16,172,16.0.0/12
EOF
exit
Next make sure that all nodes have this file, we’ll use an image update.
[root@headnode ~]# cmsh
[headnode]% device
[headnode->device]% imageupdate -c default -w
... wait for it to succeed ...
Since we chose an image update to avoid a reboot, we’ll have to reload systemd and restart containerd. We’ll use pdsh
for this.
# for containerd
pdsh -g category=default "systemctl daemon-reload && systemctl restart containerd"
# for docker
pdsh -g category=default "systemctl daemon-reload && systemctl restart docker"
At this point containerd
or docker
itself should use the proxy for tasks such as pulling container images from external domains. Now you can continue with the next section in order to configure the same for inside containers.
5. Some note on Docker
There appear to be instructions for configuring HTTP proxies in docker on this URL at the time of writing: https://docs.docker.com/network/proxy. However, please note that this does not seem to be for the same use-case, this appears to be for users of the docker
commandline tool.
6. Configure default Kubernetes container environment using PodPreset Webhook
We will use the following project: Red Hat’s PodPreset Based Webhook.
We will run the installation steps from the Head Node.
# load module file for Kubernetes if not already done so
module load kubernetes/default/1.21.4
# install dependencies to build the project
yum install go # installs golang-go-latest
# install cert-manager in k8s, if it doesn't already exist, for example:
kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.8.2/cert-manager.yaml
# checkout the project and build it
git clone https://github.com/redhat-cop/podpreset-webhook
cd podpreset-webhook
make deploy IMG=quay.io/redhat-cop/podpreset-webhook:latest
# create a PodPreset for our proxy
cat << EOF > podpreset.yaml
apiVersion: redhatcop.redhat.io/v1alpha1
kind: PodPreset
metadata:
name: proxy-environment-variables
spec:
env:
- name: http_proxy
value: http://myuser:password@1.2.3.4:3128
- name: https_proxy
value: http://myuser:password@1.2.3.4:3128
- name: no_proxy
value: 10.0.0.0/8,192.168.0.0/16,172,16.0.0/12
EOF
kubectl create -f podpreset.yaml
Now it’s possible to continue to the next section to validate if the proxy works
7. Verify if the configured proxy is working correctly
Now we should be able to test if this worked. We will see if the environment is set inside a running Pod, and if it’s used by wget
.
[root@headnode ~]# kubectl run test -it --rm --restart=Never --image=busybox:latest /bin/sh
If you don't see a command prompt, try pressing enter.
/ # env | grep proxy
no_proxy=10.0.0.0/8,192.168.0.0/16,172,16.0.0/12
https_proxy=http://myuser:password@1.2.3.4:3128
http_proxy=http://myuser:password@1.2.3.4:3128
/ # wget -O - -S http://ifconfig.co
Connecting to 10.3.191.203:3128 (10.3.191.203:3128)
HTTP/1.1 200 OK
Date: Wed, 29 Jun 2022 09:36:21 GMT
Content-Type: text/plain; charset=utf-8
Content-Length: 14
CF-Cache-Status: DYNAMIC
Report-To: ...
Server: cloudflare
CF-RAY: 722dad638e110bc1-AMS
alt-svc: h3=":443"; ma=86400, h3-29=":443"; ma=86400
X-Cache: MISS from ...
X-Cache-Lookup: HIT from ...:3128
Connection: close
writing to stdout
...
- 100% |******************************************************************************************************************************************************************************************************************************************| 14 0:00:00 ETA
written to stdout
/ # exit
pod "test" deleted