How do I change the monitoring data backup location on my HA cluster?

The following steps can be used to change the monitoring data backup location on an HA cluster.

NOTE: The following steps are only for NVIDIA Base Command Manager 10 and up, and Bright Cluster Manager 9.0 and up. The steps will require restarting the cmd (CMDaemon) service on both headnodes, it is therefore recommended to schedule an outage window to perform the steps.

1) Suspend the monitoring subsystem for both headnodes by running the following set of commands on the active headnode (so that existing backup data is not modified while copying the existing monitoring backup data to the new location):

# cmsh
% device
% foreach -t headnode (monitoringsuspend)
% quit

2) On each headnoode, copy the monitoring backup data directory for the other headnode to the new desired location:

# ls -l /var/spool/cmd/backup/
# cp -ax /var/spool/cmd/backup/<other_headnode_hostname> /path/to/new/location/

3) Set the new backup location in cmsh for each headnode by running the following set of commands on the active headnode as follows:

# cmsh
% device use <primary-headnode-hostname>
% roles
% use backup
% get directory
% set directory /path/to/new/location
% commit
% device use <secondary-headnode-hostname>
% set directory /path/to/new/location
% commit
% quit

4) Resume the monitoring subsystem for both headnodes by running the following set of commands on the active headnode:

# cmsh
% device
% foreach -t headnode (monitoringresume)
% quit

5) Restart the cmd service on both headnodes by running the following command on each headnode, first on the active headnode, then on the passive headnode:

# systemctl restart cmd