How do I clone the monitoring data database?

In Bright 7.x the monitoring data is a database of the results of healthcheck and metric monitoring. CMDaemon deals with this, as well as the regular CMDaemon database and the Slurm database.

The dbreclone option for the cmha command will normally clone the CMDaemon database from the current head node to the passive head node. The option also clones the Slurm database.

It does not however clone the monitoring database, because that tends to be unfeasibly large and historical, and is not required for getting Bright up and running.

To clone the CMDaemon monitoring database, you can follow these steps:

1. stop CMDaemon on both head nodes:

[root@head1 ~]# service cmd stop
Waiting for CMDaemon (7457) to terminate...
[ OK ]
[root@head2 ~]# service cmd stop
Waiting for CMDaemon (7457) to terminate...
[ OK ]

2. install the netcat (nc) tool on both head nodes, if it is not installed already (for RHEL 7 the package name is nmap-ncat):

[root@head1 ~]# yum install nc[root@head2 ~]# yum install nc

3. on the passive head node, start listening on a random unused port and pipe received contents to the MySQL cmdaemon_mon database:

[root@head2 ~]# nc -l 9999 | mysql -u$(grep DBUser /cm/local/apps/cmd/etc/cmd.conf | gawk '{print $3}' | sed 's#\"##g') -p$(grep DBPass /cm/local/apps/cmd/etc/cmd.conf | gawk '{print $3}' | sed 's#\"##g') cmdaemon_mon

4. on the active head node, dump the MySQL cmdaemon_mon database and pipe the contents with nc to the passive head node through the same random unused port which was chosen in the previous step:

[root@head1 ~]# time mysqldump -u$(grep DBUser /cm/local/apps/cmd/etc/cmd.conf | gawk '{print $3}' | sed 's#\"##g') -p$(grep DBPass /cm/local/apps/cmd/etc/cmd.conf | gawk '{print $3}' | sed 's#\"##g') cmdaemon_mon | nc head2 9999

5. start CMDaemon on both head nodes:

[root@head1 ~]# service cmd start
Waiting for CMDaemon to start...                           [  OK  ]
[root@head2 ~]# service cmd start
Waiting for CMDaemon to start...                           [  OK  ]