GPFS can be integrated with Bright in two ways. Either by mounting a GPFS filesystem on an existing Bright Cluster, or by adding GPFS cluster functionality to the Bright Cluster.
A. Mounting an External GPFS filesystem (GPFS over NFS)
An external GPFS filesystem can be mounted on a Bright Cluster via NFS by carrying out the following steps:
A1. Export the GPFS filesystem using NFS
- Edit /etc/exports and add an entry to export the GPFS filesystem to the network of the Bright Cluster:
/gpfs1/test <base network ip>/<netmask>(rw,fsid=745,no_root_squash,async)
- Make sure that the clocks of all nodes in the GPFS cluster are synchronized. If this is not done, then NFS access to the data, as well as other GPFS file system operations, may be disrupted.
- Restart NFS, and ensure that it is properly configured and running.
- Ensure that the firewall on the GPFS cluster will accept the incoming connections for NFS from the network that the Bright Cluster is on.
A2. Mount the GPFS filesystem on the Bright Cluster as NFS (when NSD is not directly attached to the nodes)
a. For the nodes that are supposed to mount the GPFS filesystem: inside their software image, make a new directory to mount the GPFS filesystem:
# mkdir /cm/images/default-image/gpfs1/test
b. Add an fsmount entry for the nodes (head and compute nodes) which are supposed to mount the GPFS filesystem:
for the head node:
# cmsh % device fsmounts master % add /gpfs1 % set device gpfs-test:/gpfs1/test % set filesystem nfs % commit % !mount -a
for the compute nodes:
% category fsmounts default % add /gpfs1 % set device gpfs-test:/gpfs1/test% set filesystem nfs % commit
A3. Mount the GPFS filesystem on the Bright Cluster using the mmmount command (when NSD is directly attached to the nodes)
Follow the steps in Section B “Adding GPFS Cluster Functionality to Bright Cluster” section without creating the NSD (steps B7 and B8). Also step B4 can be skipped, as the cluster will already be created. You can use the mmaddnode command on the GPFS cluster to add the new nodes to the GPFS cluster.
B Adding GPFS Cluster Functionality to Bright Cluster
You can follow this brief guide to installing the GPFS file system on a RedHat-like system, and integrating it with Bright:
B1. Installing The GPFS Packages
- On the headnodes:
Run the installer and accept the license
# ./Spectrum_Scale_Protocols_Advanced-4.2.1.0-x86_64-Linux-install Extracting License Acceptance Process Tool to /usr/lpp/mmfs/4.2.1.0 ... tail -n +563 ./Spectrum_Scale_Protocols_Advanced-4.2.1.0-x86_64-Linux-install | tar -C /usr/lpp/mmfs/4.2.1.0 -xvz --exclude=installer --exclude=*_rpms --exclude=*rpm --exclude=*tgz --exclude=*deb 1> /dev/null Installing JRE ... tail -n +563 ./Spectrum_Scale_Protocols_Advanced-4.2.1.0-x86_64-Linux-install | tar -C /usr/lpp/mmfs/4.2.1.0 --wildcards -xvz ibm-java*tgz 1> /dev/null tar -C /usr/lpp/mmfs/4.2.1.0/ -xzf /usr/lpp/mmfs/4.2.1.0/ibm-java*tgz Defaulting to --text-only mode. Invoking License Acceptance Process Tool ... /usr/lpp/mmfs/4.2.1.0/ibm-java-x86_64-71/jre/bin/java -cp /usr/lpp/mmfs/4.2.1.0/LAP_HOME/LAPApp.jar com.ibm.lex.lapapp.LAP -l /usr/lpp/mmfs/4.2.1.0/LA_HOME -m /usr/lpp/mmfs/4.2.1.0 -s /usr/lpp/mmfs/4.2.1.0 -text_only LICENSE INFORMATION The Programs listed below are licensed under the following License Information terms and conditions in addition to the Program license terms previously agreed to by Client and IBM. If Client does not have previously agreed to license terms in effect for the Program, the International Program License Agreement (Z125-3301-14) applies. Program Name: IBM Spectrum Scale V4.2.1 Program Number: 5641-GPF Program Name: IBM Spectrum Scale V4.2.1 Program Number: 5725-Q01 Press Enter to continue viewing the license agreement, or enter "1" to accept the agreement, "2" to decline it, "3" to print it, "4" to read non-IBM terms, or "99" to go back to the previous screen. 1 License Agreement Terms accepted. Extracting Product RPMs to /usr/lpp/mmfs/4.2.1.0 ... tail -n +563 ./Spectrum_Scale_Protocols_Advanced-4.2.1.0-x86_64-Linux-install | tar -C /usr/lpp/mmfs/4.2.1.0 --wildcards -xvz installer object_rpms ganesha_rpms gpfs_rpms rhel6 sles11 sles12 smb_rpms zimon_rpms manifest 1> /dev/null - installer - object_rpms - ganesha_rpms - gpfs_rpms - rhel6 - sles11 - sles12 - smb_rpms - zimon_rpms - manifest Removing License Acceptance Process Tool from /usr/lpp/mmfs/4.2.1.0 ... rm -rf /usr/lpp/mmfs/4.2.1.0/LAP_HOME /usr/lpp/mmfs/4.2.1.0/LA_HOME Removing JRE from /usr/lpp/mmfs/4.2.1.0 ... rm -rf /usr/lpp/mmfs/4.2.1.0/ibm-java*tgz ================================================================== Product rpms successfully extracted to /usr/lpp/mmfs/4.2.1.0 Cluster installation and protocol deployment To install a cluster or deploy protocols with the Spectrum Scale Install Toolkit: /usr/lpp/mmfs/4.2.1.0/installer/spectrumscale -h To install a cluster manually: Use the gpfs rpms located within /usr/lpp/mmfs/4.2.1.0/gpfs_rpms To upgrade an existing cluster using the Spectrum Scale Install Toolkit: 1) Copy your old clusterdefinition.txt file to the new /usr/lpp/mmfs/4.2.1.0/installer/configuration/ location 2) Then run: /usr/lpp/mmfs/4.2.1.0/installer/spectrumscale upgrade -h To add nodes to an existing cluster using the Spectrum Scale Install Toolkit: 1) Add nodes to the clusterdefinition.txt file: /usr/lpp/mmfs/4.2.1.0/installer/spectrumscale node add -h 2) Install GPFS on the new nodes: /usr/lpp/mmfs/4.2.1.0/installer/spectrumscale install -h 3) Deploy protocols on the new nodes: /usr/lpp/mmfs/4.2.1.0/installer/spectrumscale deploy -h To add NSDs or file systems to an existing cluster using the Spectrum Scale Install Toolkit: 1) Add nsds and/or filesystems with: /usr/lpp/mmfs/4.2.1.0/installer/spectrumscale nsd add -h 2) Install the NSDs: /usr/lpp/mmfs/4.2.1.0/installer/spectrumscale install -h 3) Deploy the new file system: /usr/lpp/mmfs/4.2.1.0/installer/spectrumscale deploy -h ================================================================================== To get up and running quickly, visit our wiki for an IBM Spectrum Scale Protocols Quick Overview: https://www.ibm.com/developerworks/community/wikis/home?lang=en#!/wiki/General%20Parallel%20File%20System%20%28GPFS%29/page/Protocols%20Quick%20Overview%20for%20IBM%20Spectrum%20Scale ===================================================================================RPMs are now available in /usr/lpp/mmfs/4.2.1.0/gpfs_rpms
# cd /usr/lpp/mmfs/4.2.1.0/gpfs_rpms# rpm -ivh gpfs.base-4.2.1-0.x86_64.rpm gpfs.docs-4.2.1-0.noarch.rpm gpfs.gpl-4.2.1-0.noarch.rpm gpfs.msg.en_US-4.2.1-0.noarch.rpm gpfs.gskit-8.0.50-57.x86_64.rpm
- On the compute nodes:
The best way to install GPFS RPMs on the regular nodes is to install them inside the software image. This guarantees that the packages persist on the nodes, and will not be erased after a sync update, a reboot, or even a full install.
# rpm --root /cm/images/default-image/ -ivh gpfs.base-4.2.1-0.x86_64.rpm gpfs.docs-4.2.1-0.noarch.rpm gpfs.gpl-4.2.1-0.noarch.rpm gpfs.msg.en_US-4.2.1-0.noarch.rpm gpfs.gskit-8.0.50-57.x86_64.rpm
The directory default-image in the preceding can be substituted with the appropriate software image name.
Notes:
- After installing the packages, you should append the path of GPFS binaries to the default path of the binaries:
# export PATH=$PATH:/usr/lpp/mmfs/bin
- The export statement could be added to
/etc/profile.d/gpfs.shon the head node, and also inside the software image (for example/cm/images/default-image/etc/profile.d/gpfs.sh) so that the configuration becomes permanent
B2. Configure the node category exclude lists
After installing the packages, the /var/mmfs path should be added to the following lists as items to be excluded. This prevents the GPFS configuration files from being erased.
# cmsh % category use default
% set excludelistsyncinstall (add the following line) no-new-files: - /var/mmfs
% set excludelistgrab (add the following line) - /var/mmfs
% set excludelistgrabnew (add the following line) - /var/mmfs
% set excludelistupdate (add the following line) no-new-files: - /var/mmfs % commitThe default category can be substituted with the appropriate category name.
Notes:
- The
/var/mmfsshouldn’t be added to the full exclude list (excludelistfullinstall). This is because provisioning a node in FULL mode will re-partition the hard drives and will re-create the filesystem, and will then synchronize the image, so that/var/mmfswill anyway already be destroyed on the node. - In case a node was provisioned in a FULL install mode, the node should be re-added to the GPFS cluster.
B3. Building The Compatibility Layer
Before starting GPFS, the GPFS compatibility layer must be built. This layer is a set of binaries that need to be built locally from source code, to match the Linux kernel and configuration of the hosts. To build the layer:
# cd /usr/lpp/mmfs/src # make LINUX_DISTRIBUTION=REDHAT_AS_LINUX Autoconfig # make World # make InstallImages # make rpm
The “make rpm” step generates an RPM package for portability binaries, so that the compatibility layer can conveniently be deployed on other machines with an identical architecture, distribution level, and Linux kernel. (Ie: the binary is only “portable” to pretty much the same machines. That is because portability here means that it inserts modules into the kernel without any need to rebuild the kernel to support GPFS. Ie the administrator actually needs to restrict where it is deployed to ensure this kind of portability). The generated rpm, gpfs.gplbin*.rpm, can be installed on the software image with:
# rpm --root /cm/images/default-image -ivh /root/rpmbuild/RPMS/x86_64/gpfs.gplbin-3.10.0-514.2.2.el7.x86_64-4.2.1-0.x86_64.rpm
# rpm -ivh /root/rpmbuild/RPMS/x86_64/gpfs.gplbin-3.10.0-514.2.2.el7.x86_64-4.2.1-0.x86_64.rpms
if we are using default-image.
We also need to build the kernel modules with:
# LINUX_DISTRIBUTION=REDHAT_AS_LINUX mmbuildgpl
--------------------------------------------------------
mmbuildgpl: Building GPL module begins at Thu Oct 20 14:50:31 CEST 2016.
--------------------------------------------------------
Verifying Kernel Header...
kernel version = 20632642 (2.6.32-642.6.1.el6.x86_64, 2.6.32-642.6.1)
module include dir = /lib/modules/2.6.32-642.6.1.el6.x86_64/build/include
module build dir = /lib/modules/2.6.32-642.6.1.el6.x86_64/build
kernel source dir = /usr/src/linux-2.6.32-642.6.1.el6.x86_64/include
Found valid kernel header file under /lib/modules/2.6.32-642.6.1.el6.x86_64/build/include
Verifying Compiler...
make is present at /usr/bin/make
cpp is present at /usr/bin/cpp
gcc is present at /usr/bin/gcc
g++ is present at /usr/bin/g++
ld is present at /usr/bin/ld
Verifying Additional System Headers...
Verifying kernel-headers is installed ...
Command: /bin/rpm -q kernel-headers
The required package kernel-headers is installed
make World ...
make InstallImages ...
--------------------------------------------------------
mmbuildgpl: Building GPL module completed successfully at Thu Oct 20 14:51:58 CEST 2016.
------------------------------------------------------B4. Creating The GPFS Cluster
The command mmcrcluster is used to create the GPFS cluster. This command has two mandatory options:
- -p to specify the primary GPFS cluster configuration server
- -N to specify the nodes
To make things easier for installation, it is useful to create a file listing all of the nodes in the GPFS cluster using either host names or IP addresses. For example: create the file gpfs.allnodes, listing the nodes one per line:
gpfs-test.cm.cluster:quorumnode001node002
Then run the mmcrcluster command:
# mmcrcluster -N gpfs.allnodes -p gpfs-test.cm.cluster -r /usr/bin/ssh Warning: Permanently added 'gpfs-test.cm.cluster,10.141.255.254' (RSA) to the list of known hosts. Thu May 23 17:18:40 CEST 2013: mmcrcluster: Processing node gpfs-test.cm.cluster Thu May 23 17:18:43 CEST 2013: mmcrcluster: Processing node node001.cm.cluster mmcrcluster: Command successfully completed mmcrcluster: Warning: Not all nodes have proper GPFS license designations. Use the mmchlicense command to designate licenses as needed. mmcrcluster: Propagating the cluster configuration data to all affected nodes. This is an asynchronous process. The -r option specifies the absolute path to the remote shell program. By default, rsh is used, but that is a bad idea, so we choose to replace it with ssh. To add another node to the cluster after the initial creation, the mmaddnode command is used.
After creating the cluster with mmrcluster, or after adding a node with mmaddnode, the mmfsEnvLevel1, mmfsNodeData, and mmsdrfs files get created under /var/mmfs/gen. These files are necessary for node identification to the GPFS server. A FULL provisioning of the nodes recreates the partitions and the filesystem of the nodes, so the configurations stored under /var/mmfs will be destroyed. In this case, the fully provisioned node should be re-added to the cluster. This can be done by rebooting the node to be unpingable for a few moments and then removing it with the mmdelnode command, and re-adding it with the mmaddnode command:
# mmdelnode -N node001.cm.cluster Verifying GPFS is stopped on all affected nodes ... mmdelnode: Command successfully completed # mmaddnode -N node001.cm.cluster Fri Nov 22 15:57:30 CET 2013: mmaddnode: Processing node node001.cm.cluster mmaddnode: Command successfully completed mmaddnode: Warning: Not all nodes have proper GPFS license designations. Use the mmchlicense command to designate licenses as needed. mmaddnode: Propagating the cluster configuration data to all affected nodes. This is an asynchronous process.B5. Assign The GPFS License
The mmchlicense command is used to assign the appropriate GPFS license to each of the nodes in the cluster.
# mmchlicense server -N gpfs-test.cm.cluster
The following nodes will be designated as possessing GPFS server licenses:
gpfs-test.cm.cluster
Please confirm that you accept the terms of the GPFS server Licensing Agreement. The full text can be found at www.ibm.com/software/sla
Enter "yes" or "no": yes
mmchlicense: Command successfully completed
The “server option can be changed to client according to the role of the node in the cluster.
B6. Start The GPFS Cluster
Start GPFS by issuing the mmstartup command.
# mmstartup Fri May 24 17:08:01 CEST 2013: mmstartup: Starting GPFS ...
You can add the -N option to the mmstartup command to add a comma-separated list of nodes that should be started:
# mmstartup -N gpfs-test,node001
B7. Create a Network Shared Disk
Create new disks for use in your file systems by issuing the mmcrnsd command.
# mmcrnsd -F descfile mmcrnsd: Processing disk vdc mmcrnsd: Propagating the cluster configuration data to all affected nodes. This is an asynchronous process.
After creating the NSD, the descfile will be modified and vdc will be renamed. The new name of the vdc should be used when creating a new Filesystem on vdc using the mmcrfs command.
Note
A line in descfile should be in the following format:
“DiskName:ServerList::DiskUsage:FailureGroup:DesiredName:StoragePool”
For example,
"sdb:::dataAndMetadata:5::"
After issuing the mmcrnsd command, the contents of the descfile will be altered automatically so that it can be used in creating the filesystem as follows:
# sdb:::dataAndMetadata:5:: gpfs1nsd:::dataAndMetadata:5::system
For more information about the descfile, please check the man pages of mmcrnsd
B8. Create a new Filesystem
Create a new filesystem by issuing the mmcrfs command.
# mmcrfs gpfs1nsd -F descfile -B 512K -Q yes -T /gpfs1The following disks of gpfs1nsd will be formatted on node gpfs-test:gpfs1nsd: size 96468992 KBFormatting file system ...Disks up to size 834 GB can be added to storage pool 'system'.Creating Inode FileCreating Allocation MapsCreating Log FilesClearing Inode Allocation MapClearing Block Allocation MapFormatting Allocation Map for storage pool 'system'Completed creation of file system /dev/gpfs1nsd.
mmcrfs: Propagating the cluster configuration data to all affected nodes. This is an asynchronous process.
Notes:
- The -T option specifies the mount point directory of the GPFS file system. If it is not specified, the mount point will be set to DefaultMountDir/Device. The default value for DefaultMountDir is /gpfs but, it can be changed with the mmchconfig command.
- The -Q option activates quotas automatically when the file system is mounted. The default is no. Issue the mmdefedquota command to establish default quota values. Issue the mmedquota command to establish explicit quota values.
B9. Mount the Filesystem on the Head Node
Assuming that the head node is directly attached to the NSD, mount the newly created filesystem on it by issuing the mmmount command:
# mmmount gpfs1nsd -a Fri May 24 17:26:07 CEST 2013: mmmount: Mounting file systems ...
The mmmount command will append an entry in /etc/fstab for the mounted GPFS filesystem if it doesn’t exist:
# cat /etc/fstab [...] /dev/gpfs1nsd /gpfs1 gpfs rw,mtime,atime,quota=userquota;groupquota;filesetquota,dev=gpfs1nsd,noauto 0 0
Notes:
- The -a option in mmmount command will mount the not only the head node, but on every node attached directly to the NSD. Other nodes which are not attached directly to the mmmount command will fail.
- The filesystem will be mounted on the mount point specified with the -T option when creating the filesystem with mmcrfs command. If no value is specified, then the mount point will be what is defined by DefaultMountDir, which is “/gpfs by default. This DefaultMountDir can be changed with the mmchconfig command.
For more information, please refer to GPFS: Administration and Programming Reference.
B10. Auto Mount GPFS Filesystem on Reboot
The mmchconfig command can be used to configure GPFS to startup automatically on reboot:
# mmchconfig autoload=yes mmchconfig: Command successfully completed mmchconfig: Propagating the cluster configuration data to all affected nodes. This is an asynchronous process.
The -N option can be used to specify the node to be configured.
B11. Mount the GPFS filesystem on regular nodes
Assuming that the regular nodes are attached to the NSD via the head node.
- Make a new directory to mount the GPFS filesystem on it, inside the software image of the nodes which are supposed to mount the GPFS filesystem:
# mkdir /cm/images/default-image/gpfs1/test
- Add an fsmount entry on the nodes which are supposed to mount the GPFS filesystem:
# cmsh % category fsmounts default % add /gpfs1 % set device gpfs-test:/gpfs1/test % set filesystem nfs % commit % !mount -a
Assuming that the regular nodes are attached to the NSD directly, the mmmount command will append an entry in/etc/fstab for the mounted GPFS filesystem, if it doesn’t exist, and by default the filesystems of type gpfs get excluded by Bright so that they won’t be touched by an imageupdate command:
# cat /etc/fstab [...] /dev/gpfs1nsd /gpfs1 gpfs rw,mtime,atime,quota=userquota;groupquota;filesetquota,dev=gpfs1nsd,noauto 0 0
Notes:
- The node-installer checks the disk layout XML schema and mounts the the filesystems specified in it, but it will not mount what is specified in /etc/fstab or what is defined in the fsmounts of the category or the node. So rebooting a node will not affect the GPFS filesystem as it will not be mounted at this stage, since it is not part of the disksetup XML schema.
- By default Bright excludes filesystems of type gpfs so that they won’t get wiped by an “image update” and as a result, adding the GPFS mount points to the exclude lists is not needed.
C. Troubleshooting issues when doing these steps
Issue:
# mmcrcluster -N gpfs.allnodes -p gpfs-test.cm.cluster mmdsh: rsh: gpfs-test.cm.cluster /usr/lpp/mmfs/bin/mmremote mmrpc:1:1:1302:mmrc_checkNewClusterNode_gpfs-test_22256_1367600794_: checkNewClusterNode lc/lc2 gpfs-test.cm.cluster _NOSECONDARY_ %%home%%:20_MEMBER_NODE::0:1:gpfs-test:10.141.255.254:gpfs-test.cm.cluster:client::::::gpfs-test.cm.cluster:gpfs-test::::Q:::::: _DEFAULT_ _DEFAULT_ 14028639671563841178:lc2: No such file or directory mmdsh: rsh: gpfs-test.cm.cluster /usr/lpp/mmfs/bin/mmremote mmrpc:1:1:1302:: getRc mmrc_checkNewClusterNode_gpfs-test_22256_1367600794_: No such file or directory mmcrcluster: Unexpected error from checkNewClusterNode gpfs-test.cm.cluster. Return code: 2 mmcrcluster: Command failed. Examine previous error messages to determine cause.Resolution:
Use the “-r” option with the absolute path to ssh:
# mmcrcluster -N gpfs.allnodes -p gpfs-test.cm.cluster -r /usr/bin/ssh
Issue:
# make Autoconfig
[…]
Can’t determine the distribution type. /etc/redhat-release is present, but the release name is not recognized. Please specify the distribution type explicitly.
Workaround:
Specify the appropriate Linux distribution:
# make LINUX_DISTRIBUTION=REDHAT_AS_LINUX Autoconfig
Issue:
# mmmount gpfs1nsd Fri Nov 22 13:41:31 CET 2013: mmmount: Mounting file systems ... mmremote: GPFS is not ready to handle commands yet. mmmount: Command failed. Examine previous error messages to determine cause.
Resolution:
GPFS needs to be started on the node:
# mmstartup
Issue:
# mmstartup mmstartup: Required service not applied. Install GPFS 3.5.0.1 or later. mmstartup: Command failed. Examine previous error messages to determine cause.
Workaround:
# touch /var/mmfs/gen/IGNOREPTF1