How do I extend an OpenStack cluster with extra controller/network nodes?

This article is being updated. Please be aware the content herein, not limited to version numbers and slight syntax changes, may not match the output from the most recent versions of Bright. This notation will be removed when the content has been updated.

Scenario:

You have installed OpenStack with one controller. Now you want to add two more controllers to expand the database cluster.

Adding one more controller/network to your OpenStack cluster can be done by following these steps:

Decide on the nodes you want to use as controller/network nodes

Make sure the interfaces are configured on the network nodes that you want to add

For example :

If you have a node001 with two interfaces — one interface for the external OpenStack network, and the other interface for the internal network — then make sure that you match the interfaces configuration on your new nodes.

Now, append both the  OpenStackControllers and OpenStackNetworkNodes configuration overlay to the new nodes:

[am-c-02-08-t-c7u2-1->configurationoverlay]% append openstackcontrollers nodes node002,node003
[am-c-02-08-t-c7u2-1->configurationoverlay*]% append openstacknetworknodes nodes node002,node003
[am-c-02-08-t-c7u2-1->configurationoverlay*]% commit

Make sure you restart both of your new nodes after you commit:

[am-c-02-08-t-c7u2-1->device]% reboot -n node002,node003

While you wait for your two nodes to be rebooted, log into your first controller/network node.

Then install xtradbackup,

https://www.percona.com/doc/percona-xtrabackup/2.2/installation/yum_repo.html

on the controller/network node.

After installation,  execute the following :

On your head node :

#cat /root/.galerarc
#ssh controller-1

[root@node001 ~]# mkdir -p /cm/shared/apps/openstack/backup/
[root@node001 ~]# innobackupex --galera-info --user=root --password=password --socket=/var/lib/mysql/mysql.sock /cm/shared/apps/openstack/backup/

The previous command will create a backup directory under /cm/shared/apps/openstack/backup/

with a name derived from the date and time of the day.

[root@node001 ~]# innobackupex -apply-log --galera-info --user=root --password=2O1711H542KV24Kg --socket=/var/lib/mysql/mysql.sock /cm/shared/apps/openstack/backup/BACKUP DIRECTORY/

First we will add a file to this backup before we stream it to the new nodes

[root@node001 2017-02-08_12-03-20]# cat xtrabackup_galera_info
#touch grastate.dat
#cat  xtrabackup_galera_info 
7c6112b1-ede7-11e6-9afb-fb7093d22da5:6336

The content of xtrabackup_galera_info should be used to construct the content of grastate.dat, which should look like this:

# GALERA saved state
version: 2.1
uuid:    7c6112b1-ede7-11e6-9afb-fb7093d22da5
seqno: 6336
safe_to_bootstrap: 0

Notice that 6336 is the number after “:” in xtrabackup_galera_info
Now we need to copy this backup to /var/lib/mysql to both of your new nodes

#ssh node002
#cd /var/lib/mysql
# rm -rf *
#rsync -avzp --progress /cm/shared/apps/openstack/backup/2017-02-08_12-03-20/ .
#chown -R mysql.mysql /var/lib/mysql

Repeat the previous steps to also rsync it over to your third node

When done, go back to your head node and execute the following :

[root@am-c-02-08-t-c7u2-1 ~]# cmsh
[am-c-02-08-t-c7u2-1]% configurationoverlay
[am-c-02-08-t-c7u2-1->configurationoverlay]% append galeranodes nodes node002..node003
[am-c-02-08-t-c7u2-1->configurationoverlay*]% commit

To confirm the controllers are there, please log into any of your new controllers and execute the following:

[root@node001 ~]# mysql -uroot -ppassword
MariaDB [(none)]> show status like 'wsrep_cluster_size';

Your output should be :

+--------------------+-------+
| Variable_name      | Value |
+--------------------+-------+
| wsrep_cluster_size | 3     |
+--------------------+-------+
1 row in set (0.00 sec)