Categories

ID #1168

How do I create an InfiniBand subinterface?

How do I create an IB subinterface?

 

Subinterfaces divide primary (parent) interfaces to provide traffic isolation. Partition assignments distinguish subinterfaces from parent interfaces. Subinterface creation and removal is discussed in the Linux kernel documentation (see https://www.kernel.org/doc/Documentation/infiniband/ipoib.txt).

 

Method 1

 

1. Make sure that the software image which is used to provision the nodes has the required kernel modules to load the IB interface on startup and add them if these modules are missing:

 

# cmsh

% softwareimage kernelmodules <software-image>

% list

[...]

ib_ipoib

rdma_ucm

ib_ucm

ib_uverbs

ib_umad

rdma_cm

ib_cm

iw_cm

ib_addr

ib_sa

ib_qib

ib_mad

ib_core             

[...]

 

2. Create the partition of the InfiniBand interface first by adding the value of the partition key to the initialize script of the category to which the nodes belong:

 

# cmsh

% category use <category name>

% set initializescript

echo 0x8002 > /sys/class/net/ib0/create_child

% commit

 

3. Reboot the nodes for the changes to take effect and use “ifconfig -a” command after the nodes are UP again to verify that the interface was created successfully:

 

[root@node001 ~]# ifconfig

[...]

ib0.8002  Link encap:InfiniBand  HWaddr 80:00:00:05:FE:80:00:00:00:00:00:00:00:00:00:00:00:00:00:00  

         UP BROADCAST RUNNING MULTICAST  MTU:4092  Metric:1

         RX packets:0 errors:0 dropped:0 overruns:0 frame:0

         TX packets:0 errors:0 dropped:0 overruns:0 carrier:0

         collisions:0 txqueuelen:256

         RX bytes:0 (0.0 b)  TX bytes:0 (0.0 b)

[...]

 

Method 2

 

The child interface can also be created by adding an echo statement to rc.local in the software image of the nodes. In this case ,adding the kernel modules to the software image will not be necessary.

 

# vim /cm/images/cuda-rc/etc/rc.local

[...]

echo 0x8002 > /sys/class/net/ib0/create_child

[...]

 

 

Tags: -

Related entries:

You cannot comment on this entry