How do I create an InfiniBand subinterface?

This article is being updated. Please be aware the content herein, not limited to version numbers and slight syntax changes, may not match the output from the most recent versions of Bright. This notation will be removed when the content has been updated.

Subinterfaces divide primary (parent) interfaces to provide traffic isolation. Partition assignments distinguish subinterfaces from parent interfaces. Subinterface creation and removal is discussed in the Linux kernel documentation (see https://www.kernel.org/doc/Documentation/infiniband/ipoib.txt).

Method 1

1. Make sure that the software image which is used to provision the nodes has the required kernel modules to load the IB interface on startup and add them if these modules are missing:
# cmsh
% softwareimage kernelmodules <software-image>
% list
[...]
ib_ipoib
rdma_ucm
ib_ucm
ib_uverbs 
ib_umad
rdma_cm
ib_cm
iw_cm
ib_addr
ib_sa
ib_qib
ib_mad
ib_core             
[...]

2. Create the partition of the InfiniBand interface first by adding the value of the partition key to the initialize script of the category to which the nodes belong:

# cmsh
% category use <category name>
% set initializescript
#!/bin/bash
echo 0x8002 > /sys/class/net/ib0/create_child
% commit

3. Reboot the nodes for the changes to take effect and use “ifconfig -a” command after the nodes are UP again to verify that the interface was created successfully:
[root@node001 ~]# ifconfig
[...]
ib0.8002  Link encap:InfiniBand  HWaddr 80:00:00:05:FE:80:00:00:00:00:00:00:00:00:00:00:00:00:00:00  

         UP BROADCAST RUNNING MULTICAST  MTU:4092  Metric:1

         RX packets:0 errors:0 dropped:0 overruns:0 frame:0

         TX packets:0 errors:0 dropped:0 overruns:0 carrier:0

         collisions:0 txqueuelen:256

         RX bytes:0 (0.0 b)  TX bytes:0 (0.0 b)

[...]

Method 2

The child interface can also be created by adding an echo statement to rc.local in the software image of the nodes. In this case ,adding the kernel modules to the software image will not be necessary.
# vim /cm/images/cuda-rc/etc/rc.local
[...]
echo 0x8002 > /sys/class/net/ib0/create_child
[...]