1. Home
  2. How to configure an HTTP(s) proxy for Kubernetes

How to configure an HTTP(s) proxy for Kubernetes

1. Prerequisites
  • This article was written with 9.2 in mind, but the content should apply for older versions as well.
  • Kubernetes is deployed with either docker or containerd.
  • We’re assuming all the Kubernetes nodes do not have external internet access without a configured HTTP(s) proxy.
2. Issues potentially addressed by this KB article
  1. Kubernetes cannot pull container images (and because of it cannot create containers for Pods)

    We will make the container runtime use an http proxy.
  2. The software running inside the containers do not have internet access.

    We will expose http proxy environment variables inside the container.

Please note that setting these environment variables does not guarantee that they will actually be used by the software. Although it’s common practice that many tools adhere to these environment variables, not all tools might implement this feature.

3. Figure out the correct environment variables

These are probably known to your organization, and something similar to:

http_proxy=http://myuser:password@1.2.3.4:3128
https_proxy=http://myuser:password@1.2.3.4:3128
no_proxy=10.0.0.0/8,192.168.0.0/16,172,16.0.0/12
4. Configuring the proxy for the containerd or docker service

For the sake of simplicity of this KB article we have only one category to deal with. Please repeat these steps for Head Nodes in case they are involved, or any additional categories that may be involved.

First the steps for containerd. The steps for docker are almost the same.

# chroot into the software image

cm-chroot-sw-img /cm/images/default-image/

# create directory to put an override file for containerd
mkdir -p /etc/systemd/system/containerd.service.d

# create the file
cat << EOF > /etc/systemd/system/containerd.service.d/override.conf
[Service]
Environment=http_proxy=http://myuser:password@1.2.3.4:3128
Environment=https_proxy=http://myuser:password@1.2.3.4:3128
Environment=no_proxy=10.0.0.0/8,192.168.0.0/16,172,16.0.0/12
EOF

# exit the chroot
exit

For docker use:

cm-chroot-sw-img /cm/images/default-image/
mkdir -p /etc/systemd/system/docker.service.d
cat << EOF > /etc/systemd/system/docker.service.d/override.conf
[Service]
Environment=http_proxy=http://myuser:password@1.2.3.4:3128
Environment=https_proxy=http://myuser:password@1.2.3.4:3128
Environment=no_proxy=10.0.0.0/8,192.168.0.0/16,172,16.0.0/12
EOF
exit

Next make sure that all nodes have this file, we’ll use an image update.

[root@headnode ~]# cmsh
[headnode]% device 
[headnode->device]% imageupdate -c default -w
... wait for it to succeed ...

Since we chose an image update to avoid a reboot, we’ll have to reload systemd and restart containerd. We’ll use pdsh for this.

# for containerd
pdsh -g category=default "systemctl daemon-reload && systemctl restart containerd"

# for docker
pdsh -g category=default "systemctl daemon-reload && systemctl restart docker"

At this point containerd or docker itself should use the proxy for tasks such as pulling container images from external domains. Now you can continue with the next section in order to configure the same for inside containers.

5. Some note on Docker

There appear to be instructions for configuring HTTP proxies in docker on this URL at the time of writing: https://docs.docker.com/network/proxy. However, please note that this does not seem to be for the same use-case, this appears to be for users of the docker commandline tool.

6. Configure default Kubernetes container environment using PodPreset Webhook

We will use the following project: Red Hat’s PodPreset Based Webhook.

We will run the installation steps from the Head Node.

# load module file for Kubernetes if not already done so
module load kubernetes/default/1.21.4

# install dependencies to build the project
yum install go  # installs golang-go-latest

# install cert-manager in k8s, if it doesn't already exist, for example:
kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.8.2/cert-manager.yaml

# checkout the project and build it
git clone https://github.com/redhat-cop/podpreset-webhook
cd podpreset-webhook
make deploy IMG=quay.io/redhat-cop/podpreset-webhook:latest

# create a PodPreset for our proxy

cat << EOF > podpreset.yaml
apiVersion: redhatcop.redhat.io/v1alpha1
kind: PodPreset
metadata:
  name: proxy-environment-variables
spec:
  env:
  - name: http_proxy
    value: http://myuser:password@1.2.3.4:3128
  - name: https_proxy
    value: http://myuser:password@1.2.3.4:3128
  - name: no_proxy
    value: 10.0.0.0/8,192.168.0.0/16,172,16.0.0/12
EOF

kubectl create -f podpreset.yaml

Now it’s possible to continue to the next section to validate if the proxy works

7. Verify if the configured proxy is working correctly

Now we should be able to test if this worked. We will see if the environment is set inside a running Pod, and if it’s used by wget.

[root@headnode ~]# kubectl run test -it --rm --restart=Never --image=busybox:latest /bin/sh
If you don't see a command prompt, try pressing enter.

/ # env | grep proxy
no_proxy=10.0.0.0/8,192.168.0.0/16,172,16.0.0/12
https_proxy=http://myuser:password@1.2.3.4:3128
http_proxy=http://myuser:password@1.2.3.4:3128

/ # wget -O - -S http://ifconfig.co
Connecting to 10.3.191.203:3128 (10.3.191.203:3128)
  HTTP/1.1 200 OK
  Date: Wed, 29 Jun 2022 09:36:21 GMT
  Content-Type: text/plain; charset=utf-8
  Content-Length: 14
  CF-Cache-Status: DYNAMIC
  Report-To: ...
  Server: cloudflare
  CF-RAY: 722dad638e110bc1-AMS
  alt-svc: h3=":443"; ma=86400, h3-29=":443"; ma=86400
  X-Cache: MISS from ...
  X-Cache-Lookup: HIT from ...:3128
  Connection: close
  
writing to stdout
...
-                    100% |******************************************************************************************************************************************************************************************************************************************|    14  0:00:00 ETA
written to stdout
/ # exit
pod "test" deleted
Updated on June 29, 2022

Leave a Comment