One of my headnodes has died. How do I restore High Availability on my cluster?

This article assumes that HA is setup and configured properly, and that one of the nodes has completely died and is being replaced with new hardware.

1. Bright’s license is tied to the MAC addresses of both headnodes.  First unlock the product key via the support portal http://customer.brightcomputing.com (left hand menu)
2. Run `request-license` on the headnode
   • Re-use private key and settings from existing license? [Y/n] Y
   • Add the MAC of the Primary Headnode, Y to Secondary, add Secondary MAC
   • Submit certificate request to http://licensing.brightcomputing.com/licensing/index.cgi ? [Y/n] Y
   • Install license? [Y/n] y

3. PXE boot the new hardware, and when the Cluster Manager PXE Environment comes up, select “Rescue”

4. Login as root (no password)
   • /cm/cm-clone-install –failover
   • Enter root password
   • C Continue partition layout on disk
   • Y confirm install

When complete and prompted:  Reboot

5. From the primary headnode run cmha-setup
   • Setup
   • Finalize

When complete and prompted, Reboot again

[root@head01 log]# cmha status

Node Status: running in active mode

head01* -> head02
failoverping  [  OK  ]
mysql         [  OK  ]
ping          [  OK  ]
status        [  OK  ]

head02 -> head01*
failoverping  [  OK  ]
mysql         [  OK  ]
ping          [  OK  ]
status        [  OK  ]