Disable Calico IPIP in Bright Cluster Manager 9.2, 9.1, 9.0.

1. Prerequisites

• This KB article focuses on Bright Cluster Manager versions 9.0 >=< 9.2, however the instructions might work for older versions as well.
• IPIP is enabled (see next section to determine if it is)

IPIP can add significant overhead for this reason it’s better to disable it if possible or try other another solution where you can avoid it.

2. Determine if Calico is running with IPIP enabled or not

For this we can use calicoctl as follows.

[root@headnode ~]# module load kubernetes/default/1.21.4 
[root@headnode ~]# calicoctl get ippool -o wide
NAME                  CIDR            NAT    IPIPMODE   VXLANMODE   DISABLED   SELECTOR   
default-ipv4-ippool   172.29.0.0/16   true   Always     Never       false      all()      

The above shows IPIPMODE is enabled Always. Which means it is always enabled. The following is copied from the upstream calico documentation:

IPIP Mode to use for the IPv4 Pool created at start up. If set to a value other than Never, CALICO_IPV4POOL_VXLAN should not be set. [Default: Always]

Permitted values are: Always, CrossSubnet, Never

The CrossSubnet value will enable it only accross different subnets, and Never (or Off) will disable it.

3. First update Calico manifest in Bright Cluster Manager

When calico is first initialized, it creates a default IPPool with the name default-ipv4-ippool. Once this has been created, it won’t be re-created, unless someone might delete it and calico components, to force calico to re-initialize.

Calico initializes this IPPool based after reading a set of environmental variables passed to the calico-node containers through the YAML that defines the calico resources. These resources can be found via cmsh and edited there.

[root@headnode ~]# cmsh
[headnode]% kubernetes 
[headnode->kubernetes[default]]% appgroups 
[headnode->kubernetes[default]->appgroups]% use system 
[headnode->kubernetes[default]->appgroups[system]]% applications 
[headnode->kubernetes[default]->appgroups[system]->applications]% use calico
[headnode->kubernetes[default]->appgroups[system]->applications[calico]]% set config

The “set config” command in the above block will open an editor where all the YAML is presented. In this file we have to look for the following environment variable:

# Enable IPIP
- name: CALICO_IPV4POOL_IPIP
  value: "Always"

We have to change it from “Always” to “Never” and save the file, then proceed to with the “commit” command in cmsh, as follows.

[headnode->kubernetes*[default*]->appgroups*[system*]->applications*[calico*]]% commit

4. Update the default IPPool using calicoctl

We can use “calicoctl patch” for this purpose, as follows.

[root@headnode ~]# module load kubernetes/default/1.21.4 
[root@headnode ~]# calicoctl patch ippool default-ipv4-ippool --patch='{"spec": {"ipipMode": "Never"}}'
Successfully patched 1 'IPPool' resource

[root@headnode ~]# calicoctl get ippool -o wide
NAME                  CIDR            NAT    IPIPMODE   VXLANMODE   DISABLED   SELECTOR   
default-ipv4-ippool   172.29.0.0/16   true   Never      Never       false      all()      

No additional restarts of Calico Pods are needed for this change.

5. Reverting back to IPIPMODE Always

We have to repeat sections #3 and #4 but change everything to “Always” instead of “Never”.