Categories

ID #1295

How do I install BeeGFS?

How do I install BeeGFS?

 

Integrating BeeGFS with Bright is described here.


We use the following configuration:

  • Hosts:

    • beegfs-mgmt: Bright 7.1/RHEL7 Head Node

    • beegfs-mds: RHEL7 Server

    • beegfs-ss01: RHEL7 Server

    • beegfs-ss02: RHEL7 Server

    • beegfs-client001: Bright 7.1/RHEL7 Compute Node

    • beegfs-client002: Bright 7.1/RHEL7 Compute Node

  • BeeGFS Services:

    • 1 BeeGFS Management Server: beegfs-mgmt

    • 1 BeeGFS Metadata: beegfs-mds

    • 2 BeeGFS Storage Servers: beegfs-ss01 and beegfs-ss02

    • BeeGFS Clients: beegfs-client001 and beegfs-client002

    • 1 BeeGFS Admon Server (optional for graphical interface): beegfs-mgmt
  • Storage:

    • Storage servers with data partition formatted with xfs, mounted to "/data".

    • Metadata servers with data partition formatted with ext4, mounted to "/data".

 

Packages Download and Installation

The first step is to download the BeeGFS package repository file for the RHEL 7 distribution and store it in the directory “/etc/yum.repos.d/” on all nodes. The repository files for all distributions are available at http://www.beegfs.com/release/beegfs_2015.03/dists/

The second step is to carry out the packages installation.

How these two steps are carried out is explained next:

Management Server (on the Head Node)

Configure the yum repository and install the BeeGFS Management Service on the Head Node:

# ssh root@beegfs-mgmt

# wget -O /etc/yum.repos.d/beegfs-rhel7.repo http://www.beegfs.com/release/latest-stable/dists/beegfs-rhel7.repo

# yum install beegfs-mgmtd

 

Create the BeeGFS software image:

# cmsh

% softwareimage

% clone default-image beegfs-client-image

% commit

(wait until the copy of software image is completed)

 

Create the BeeGFS category:

# cmsh

% category

% clone default beegfs-client

% set softwareimage beegfs-client-image

% commit

 

Configure the yum repository and install the BeeGFS client packages on the software image:

# wget -O /cm/images/beegfs-client-image/etc/yum.repos.d/beegfs-rhel7.repo http://www.beegfs.com/release/latest-stable/dists/beegfs-rhel7.repo


# chroot /cm/images/beegfs-client-image/

# yum install beegfs-client beegfs-helperd beegfs-utils

Metadata Server

Configure the yum repository and install the BeeGFS Metadata Service on the Metadata Server:

# ssh root@beegfs-mds

# wget -O /etc/yum.repos.d/beegfs-rhel7.repo http://www.beegfs.com/release/latest-stable/dists/beegfs-rhel7.repo

# yum install beegfs-meta

Storage Server

Configure the yum repository and install the BeeGFS Storage Service on both Storage Server:

# ssh root@beegfs-ss01

# wget -O /etc/yum.repos.d/beegfs-rhel7.repo http://www.beegfs.com/release/latest-stable/dists/beegfs-rhel7.repo

# yum install beegfs-storage

 

# ssh root@beegfs-ss02

# wget -O /etc/yum.repos.d/beegfs-rhel7.repo http://www.beegfs.com/release/latest-stable/dists/beegfs-rhel7.repo

# yum install beegfs-storage

Admon Server (optional for graphical interface)

Install the BeeGFS Admon Service on the Head Node:

# ssh root@beegfs-mgmt

# yum install beegfs-admon

Basic Configuration

We need to do a few basic settings configuration before running the services.

Management Service (on the HEAD Node)

The management service needs to know where it can store its data. The service stores information such as connectivity data. The information that is stored does not require much storage space, and its data access is not performance-critical.

In this case we use the directory “/data/beegfs/mgmtd”:

# ssh root@beegfs-mgmt

# /opt/beegfs/sbin/beegfs-setup-mgmtd -p /data/beegfs/mgmtd

Preparing storage directory: /data/beegfs/mgmtd

* Creating format.conf file...

Updating config file: /etc/beegfs/beegfs-mgmtd.conf

* Setting storage directory in config file...

* Disabling usage of uninitialized storage directory in config file...

All done.

Metadata Service

The metadata service needs to know where it can store its data, and also needs to know where the management service is running.

In this case we use the directory “/data/beegfs/meta”:

# ssh root@beegfs-mds

# /opt/beegfs/sbin/beegfs-setup-meta -p /data/beegfs/meta -s 1 -m beegfs-mgmt

Preparing storage directory: /data/beegfs/meta

* Creating format.conf file...

* Creating server numeric ID file: /data/beegfs/meta/nodeNumID

Updating config file: /etc/beegfs/beegfs-meta.conf

* Setting management host: beegfs-mgmt

* Setting storage directory in config file...

* Disabling usage of uninitialized storage directory in config file...

* Setting usage of extended attributes to: true

All done.

Storage Service

The storage service needs to know where it can store its data and needs to know how to reach the management server.

In this case we use the directory “/data/beegfs/storage”:

# ssh root@beegfs-ss01

# /opt/beegfs/sbin/beegfs-setup-storage -p /data/beegfs/storage -s 1 -i 101 -m beegfs-mgmt

Preparing storage target directory: /data/beegfs/storage

* Creating format.conf file...

* Creating chunks directory...

* Creating buddymir directory...

* Creating target numeric ID file: /data/beegfs/storage/targetNumID

* Creating server numeric ID file: /data/beegfs/storage/nodeNumID

Updating config file: /etc/beegfs/beegfs-storage.conf

* Setting management host: beegfs-mgmt

* Appending to target directory list in config file...

* Disabling usage of uninitialized storage targets in config file...

All done.

 

# ssh root@beegfs-ss012

# /opt/beegfs/sbin/beegfs-setup-storage -p /data/beegfs/storage -s 2 -i 201 -m beegfs-mgmt

Preparing storage target directory: /data/beegfs/storage

* Creating format.conf file...

* Creating chunks directory...

* Creating buddymir directory...

* Creating target numeric ID file: /data/beegfs/storage/targetNumID

* Creating server numeric ID file: /data/beegfs/storage/nodeNumID

Updating config file: /etc/beegfs/beegfs-storage.conf

* Setting management host: beegfs-mgmt

* Appending to target directory list in config file...

* Disabling usage of uninitialized storage targets in config file...

All done.

BeeGFS Clients (on the Compute Node)

The clients need to know where the management service is running:

#  ssh root@beegfs-mgmt


# chroot /cm/images/beegfs-client-image/

# /opt/beegfs/sbin/beegfs-setup-client -m beegfs-mgmt

Updating config file: /etc/beegfs/beegfs-client.conf

* Setting management host: beegfs-mgmt

All done.

Admon Service (on the HEAD Node)

The optional admon service needs to know how to reach the management server:

# ssh root@beegfs-mgmt

# vim /etc/beegfs/beegfs-admon.conf

 

Find the option "sysMgmtdHost" and set it to "sysMgmtdHost=mater".

Service Startup

BeeGFS services can be started in an arbitrary order by using the corresponding init scripts.

Management Server (on the HEAD Node)

# ssh root@beegfs-mgmt

# systemctl start beegfs-mgmtd

Metadata Server

# ssh root@beegfs-mds

# systemctl start beegfs-meta

Storage Server

# ssh root@beegfs-ss01

# systemctl start beegfs-storage

 

# ssh root@beegfs-ss02

# systemctl start beegfs-storage

BeeGFS Clients (on the Compute Nodes)

Change the category of the BeeGFS compute nodes and reboot them

# ssh root@beegfs-mgmt

# cmsh

% device; foreach -n beegfs-client001..beegfs-client002 (set category beegfs-client)

% device; foreach -n beegfs-client001..beegfs-client002 (reboot)

 

Start the BeeGFS Client Services on the Compute Nodes

# ssh root@beegfs-client001

# systemctl start beegfs-helperd

# systemctl start beegfs-client

 

# ssh root@beegfs-client002

# systemctl start beegfs-helperd

# systemctl start beegfs-client

 Admon Server (on the HEAD Node)

# ssh root@beegfs-mgmt

# systemctl start beegfs-admon

 Add the firewall rule in shorewall for Admon Service

# ssh root@beegfs-mgmt

# vim /etc/shorewall/rules

[...]

ACCEPT   net            fw              tcp     8000
#
#LAST LINE -- ADD YOUR ENTRIES BEFORE THIS ONE -- DO NOT REMOVE

 

# systemctl restart shorewall

Check Connectivity

It is possible to check the detected network interfaces and transport protocols from a client node with the following commands:

# ssh root@beegfs-client001


# beegfs-ctl --listnodes --nodetype=meta --details


mgi-beegfs-mds [ID: 1]

  Ports: UDP: 8005; TCP: 8005

  Interfaces: eth1(TCP) eth0(TCP)


Number of nodes: 1

Root: 1


# beegfs-ctl --listnodes --nodetype=storage --details

mgi-beegfs-ss01.novalocal [ID: 1]

  Ports: UDP: 8003; TCP: 8003

  Interfaces: eth1(TCP) eth0(TCP)

mgi-beegfs-ss02 [ID: 2]

  Ports: UDP: 8003; TCP: 8003

  Interfaces: eth1(TCP) eth0(TCP)


Number of nodes: 2


# beegfs-ctl --listnodes --nodetype=client --details


6FD-56656DFE-beegfs-client002

  Ports: UDP: 8004; TCP: 0

  Interfaces: eth0(TCP)

716-56656DFE-beegfs-client001

  Ports: UDP: 8004; TCP: 0

  Interfaces: eth0(TCP)


Number of nodes: 2


# beegfs-net


mgmt_nodes

=============

beegfs-mgnt [ID: 1]

  Connections: TCP: 1 (10.141.255.254:8008);


meta_nodes

=============

mgi-beegfs-mds [ID: 1]

  Connections: TCP: 1 (10.141.0.4:8005);


storage_nodes

=============

mgi-beegfs-ss01.novalocal [ID: 1]

  Connections: TCP: 1 (10.141.0.5:8003);

mgi-beegfs-ss02 [ID: 2]

  Connections: TCP: 1 (10.141.0.6:8003);

Admon GUI

By default, TCP port 8000 will be opened by the daemon for HTTP connections.

To get the GUI, point your browser to http://<HeadNode>:8000 and download the jar-file from there.

The GUI can be started by double clicking from a file browser or by using the java -jar command (depending on your operating system and configuration).

At first start, you will be prompted to provide the hostname and port (default: 8000) of the host on which the Admon daemon is running. 

 

 

Admon Login

The login mechanism is based on two predefined users.

The user "Information" (which has the initial password "information") is only able to view statistics, whereas the user "Administrator" (which has the initial password "admin") is also able to perform administrative tasks.

It is highly recommended that the first thing you do is to log in using the administrative account and change the predefined passwords.

 

Links

http://www.beegfs.com/wiki/ManualInstallWalkThrough

http://www.beegfs.com/wiki/AdmonUsage

Tags: -

Related entries:

You cannot comment on this entry