Following document walks through the process of installing and configuring lustre clients in BC 3
1. Installing Packages
This is the first step in preparing the BC3.0 cluster to obtain required client packages.
| lustre-client-modules-dkms_2.14.0-ddn54-1_amd64.deb lustre-client-utils_2.14.0-ddn54-1_amd64.deb |
On some systems you may be required to install following packages
| $ apt install libmount-dev libjson-c-dev |
1.1 Installing Lustre Packages on compute and login nodes.
The below steps will guide through the process of installing luster package to the software image which would guarantee that the packages persist on the nodes across reboots, sync update or a full install.
- Copy existing packages to target directory in selected image. In this example we will be using default-image
| cp lustre-client-utils_2.14.0-ddn54-1_amd64.deb /cm/images/default-image/tmpcp lustre-client-modules-dkms_2.14.0-ddn54-1_amd64.deb /cm/images/default-image/tmp |
- Use cm-chroot-sw-image command to confirm packages are available in the image
|
cm-chroot-sw-img /cm/images/default-image |
- Reboot a node that is currently using default-image. Post reboot lustre packages will be available in /tmp on the host. Change to /tmp directory and install packages.
| apt install ./lustre-client-utils_2.14.0-ddn54-1_amd64.deb ./lustre-client-modules-dkms_2.14.0-ddn54-1_amd64.deb libmount-dev libjson-c-dev |
- Modprobe lustre and lnet to confirm required components are installed.
| modprobe lustre |
- Exit to commit changes to software image
- Repeat the above steps to install the packages on the required images (eg- dgx-osimage).
1.2 Configure lustre on slogin and compute nodes.
Configure lustre nets after booting the node with the new software image from above step.
- Update the lustre.conf file on the cpu and compute nodes to reflect the below settings.
“o2ib0(<ib_interface1>,<ib_interface2>)”, ib_interface1 and ib_ interface2 are the two IB ports connected to the storage IB fabric.
The below example lists the configuration on client node with IB interfaces ibp97s0f0, ibp225s0f0 connected to storage
| $ cat /etc/modprobe.d/lustre.conf options mds mds_num_threads=256 options oss oss_num_threads=256 options libcfs cpu_npartitions=32 options libcfs cpu_pattern=”” options ksocklnd peer_credits=128 options ko2iblnd peer_credits=32 options lnet networks=”o2ib0(ibp97s0f0,ibp225s0f0)” |
Step 4: Mount lustre at predetermined mount point .
eg:
| mount -t lustre 100.127.2.100@o2ib0,100.127.2.200@o2ib0:100.127.2.102@o2ib0,100.127.2.202@o2ib0:100.127.2.101@o2ib0,100.127.2.201@o2ib0:100.127.2.103@o2ib0,100.127.2.203@o2ib0:/bcmsp2 /lustre |
Step 5: Grab this software image to be deployed to rest of the hosts in the same category
eg:
| grabimage hostname -w -i default-image |
Repeat the above process for nodes with different architecture (eg- dgx nodes)
Follow instructions from Bright Admin manual for adding this FS mounts to respective node categories.