Upgrading with a Storage Node
Special considerations must be made when upgrading Bright versions with a storage node internal to the cluster:
The problem with this kind of setup is that all of the nodes except for the head node(s) must be shut down before starting the upgrade.
But filesystems from the storage node are being shared, and they need to be available to the head node for the upgrade.
To solve this, we will temporarily get rid of the dependency of the storage nodes internal to the cluster.
Let’s do it by example.
For this example, we use node003 (10.141.0.3) as a storage node. This storage node, when it is running, exports /cm/shared to the cluster from the local path of node003 at /storage/cm/shared:
Steps:
1. Shut down all of the nodes, except for the head node(s), and the storage node.
In this case, assuming we have nodes node001 and node002 as the other nodes on the cluster, we will shut down those nodes:
% cmsh[head]% device[head->device]% shutdown node001[head->device]% shutdown node002
2. On the active head node, copy /cm/shared over to /cm/shared.local:
# mkdir –p /cm/shared.local# cp –a /cm/shared/* /cm/shared.local
3. Remove the mount from Bright Cluster Manager, and then unmount the NFS filesystem /cm/shared:
Remove mount from Bright:
# cmsh[head]% device use master[head->device[head]]% fsmounts[head->device[head]->fsmounts]% remove /cm/shared[head->device[head]->fsmounts*]% commit[head->device[head]->fsmounts]% quit# umount /cm/shared
4. Bind-mount /cm/shared.local on /cm/shared
# mount --bind /cm/shared.local /cm/shared
To make it persistent, add the following line in /etc/fstab, after the autogenerated section:
# END AUTOGENERATED SECTION -- DO NOT REMOVE/cm/shared.local /cm/shared none bind 0 0
set storage node install mode to “” if it is currently set to NOSYNC
# cmsh[head]% device use node003[head->device[node003]% set installmode ""[head->device[node003*]% commit
5. Shut down your storage node
# cmsh[head]% device[head->device]% shutdown node003
6. Perform the upgrade according to the instructions in this article: https://kb.brightcomputing.com/knowledge-base/how-do-i-upgrade-to-bright-8-2/
7. Boot your storage node.
8. Mount the storage node’s shared filesystem, /cm/shared, onto /cm/shared.storage on the headnode
Check /etc/exports on node003 to make sure the filesystem is exported with rw permissions for the head node that will access it.
On the head node create /cm/shared.storage:
# mkdir –p /cm/shared.storage
The filesystem is added to fstab, after the autogenerated section:
10.141.0.3:/storage/cm/shared /cm/shared.storage nfs defaults 0 0
Mount the filesystem according to the fstab settings:
# mount -a
9. Synchronize /cm/shared.storage and /cm/shared.local
# rsync -avh /cm/shared.local/ /cm/shared.storage/
10. Now unmount the /cm/shared bind mount:
# umount /cm/shared
Remove it from /etc/fstab on the head node. i.e., remove the entry for /cm/shared.local:
/cm/shared.local /cm/shared none bind 0 0
Remove this entry in /etc/fstab on the head node too:
10.141.0.3:/storage/cm/shared /cm/shared.storage nfs defaults 0 0
11. Mount /cm/shared from the storage node to all the other nodes:
Using cmsh on the head node, mounting /cm/shared for the head node:
# cmsh[head]% device use master[head->device[head]% fsmounts[head->device[head]->fsmounts% add /cm/shared[head->device*[head*]->fsmounts*[/cm/shared*]% set device 10.141.0.3:/storage/cm/shared[head->device*[head*]->fsmounts*[/cm/shared*]% set filesystem nfs[head->device*[head*]->fsmounts*[/cm/shared*]% commit
Using cmsh on the head node, mounting /cm/shared for the regular nodes:
# cmsh[head->category[default]% category use default[head->category[default]% fsmounts[head->category[default]->fsmounts]% add /cm/shared[head->category*[default*]->fsmounts*[/cm/shared*]% set device 10.141.0.3:/storage/cm/shared[head->category*[default*]->fsmounts*[/cm/shared*]% set filesystem nfs[head->category*[default*]->fsmounts*[/cm/shared*]% commit
12. Boot up the rest of the nodes inside of your cluster