The Amazon EC2 Elastic Fabric Adapter (EFA) is an Elastic Network Adapter with additional capabilities to bypass the OS network layer, so that it provides a low-latency, reliable transport functionality between nodes in the cloud.
- Choose the supported instance type (https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/efa.html#efa-instance-types)
- Choose a Bright image with CM 9.0-11 or newer for one of the supported Linux distributions (https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/efa.html#efa-amis)
Prepare cloud nodes security group
To use EFA it is necessary to modify the security group for cloud nodes to allow all inbound and outbound traffic to and from the security group itself.
To find the ID of the security group that you will need to modify, you can use the following sequence of
[head42]% cloud [head42->cloud]% ls Name Type Username Account ID Default region Default type Default AMI -------- ----- --------- ----------- --------------- ------------- ------------- amazon ec2 userfoo 123456789 eu-central-1 m3.medium brigt-inst... amazon2 ec2 userbar 321343256 eu-west-1 m3.medium brigt-inst... ... [head42->cloud]% use amazon # Choose amazon provider [head42->cloud[amazon]]% vpcs [head42->cloud[amazon]->vpcs]% ls Name (key) Region baseAddress vpcID subnets ----------------- ------------- ------------ ------------ --------------------- vpc-eu-central-1 eu-central-1 10.42.0.0 vpc-131fb5e+ vpc-eu-central-1-p... vpc-eu-west-2 eu-west-2 10.42.0.0 vpc-32a5fa0+ vpc-eu-west-2-publ... ... [head42->cloud[amazon]->vpcs]% get vpc-eu-west-2 secgroupn # Choose VPC sg-0b525465c7fe64b7d
Now you can use this security group ID to find it in the AWS Console. Follow the instructions at https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/efa-start.html#efa-start-security to configure the security group. You will probably only need to add an outbound rule.
Prepare software image
Run the following command to open a chroot session in the software image (e.g.
default-image) that you are going to use for EFA-enabled cloud nodes:
And then follow the steps 3, 4 and 5 from the official documentation at https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/efa-start.html#efa-start-enable to
- install EFA software
- disable ptrace protection (if it is enabled by default)
- and optionally to install Intel MPI, if software you are going to use for HPC needs it.
You can keep the
aws-efa-installer directory created in the 3rd step to use
efa_test.sh script later to test whether EFA works. After the steps are carried out, remember to exit from the chroot using
NOTE: On Ubuntu, the AWS
efa_installer.sh script will try to install the distributions
environment-modules package. This package conflicts with Brights
cm-modules-init-client package. Before running the script, it is advised to patch the
efa_installer.sh script by running:
sed -i 's/environment-modules//g' efa_installer.sh
Set card type for cloud nodes
For Bright 9.2 and newer, to specify that instances have to use the EFA network interface, you need to set the
cardtype property for cloud nodes. That can be done in a way similar to the following
[head42]% device [head42->device]% foreach -c eu-west-2-cloud-node ( interfaces; set eth0 cardtype efa ) [head42->device]% commit
For older Bright versions, to specify that instances have to use the EFA network interface, you need to set
NetworkInterfaceType=efa tags for cloud nodes. That can be done in a way similar to the following
[head42]% device [head42->device]% foreach -c eu-west-2-cloud-node ( cloudsettings; append tags NetworkInterfaceType=efa ) [head42->device]% commit
Verify EFA works
To make sure that EFA works you need to run a compute node that you configured to use EFA.
Running the command
$ fi_info -p efa makes sure that the EFA software components were installed successfully (details on this can be found in the 3rd step in the official documentation at https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/efa-start.html#efa-start-enable).
To test EFA itself, you can run the test script
efa_test.sh from the
aws-efa-installer directory (see “Prepare software image”).
If you prefer not to have the efa installer in the software image you can just repeat the 3rd step to download and extract it to test EFA.
If EFA has been configured properly, then the output of
efa_test.sh command will be something like this:
[root@eu-west-2-cnode001 ~]# cd aws-efa-installer/ [root@eu-west-2-cnode001 aws-efa-installer]# ./efa_test.sh Starting server... Starting client... bytes #sent #ack total time MB/sec usec/xfer Mxfers/sec 64 10 =10 1.2k 0.03s 0.05 1251.20 0.00 256 10 =10 5k 0.00s 12.96 19.75 0.05 1k 10 =10 20k 0.00s 59.36 17.25 0.06 4k 10 =10 80k 0.00s 218.45 18.75 0.05 64k 10 =10 1.2m 0.00s 869.75 75.35 0.01 1m 10 =10 20m 0.01s 2001.29 523.95 0.00