This article is being updated. Please be aware the content herein, not limited to version numbers and slight syntax changes, may not match the output from the most recent versions of Bright. This notation will be removed when the content has been updated.
Gold Allocation Manager is an HPC resource accounting tool. It is open source, and released under a license that seems to be a BSD-style license.
The following instructions to install it were tested on Bright 7.1 with CentOS6u6
Download the latest sources:
wget -O gold.tar.gz https://sourceforge.net/projects/gold/files/latest/downloadUntar the files:
# tar -xzvf /root/tmp-install/gold.tar.gz
Configure Gold with MySQL database:
# cd gold-2.2.0.5
# ./configure --prefix=/cm/shared/apps/gold/2.2.0.5 --with-db=mysqlCompile the sources:
To compile the program:
# make
To install the Gold web GUI:
# make gui
Installation:
To install Gold
# make install
To install the Gold web GUI
# make install-gui
** To delete the files created by the Gold installation, you can use ‘make uninstall’.
You will also need to generate a secret key which enables secure communication between clients and server. This key is a passphrase consisting of up to 80 characters and can include spaces and the regular visible ASCII characters. Note that if you are using Gold with Moab or the Maui Scheduler, they will need both need to use a shared secret key.
# make auth_key
Enter your secret key (up to 80 characters and can include spaces): mysecret
Install the required Perl modules:
# yum install perl-Log-Log4perl.noarch# yum install perl-Config-Properties.noarch# yum install perl-Error.noarch# yum install perl-Crypt-CBC.noarch# yum install perl-Digest-SHA1.x86_64# yum install perl-Digest-HMAC.noarch# yum install perl-suidperl.x86_64
** for list of Perl Modules: http://docs.adaptivecomputing.com/gold/x385.php
Create a Gold database:
# mysql -u root -pmysql> create database gold;
Add database user for Gold:
# mysql -u root -pmysql> use mysql;mysql> create user gold;mysql> grant all on *.* to 'gold'@'localhost';
Initialize the database:
# mysql -u root -p gold < bank.sql
Edit the configuration files and add the database information:
# vim /cm/shared/apps/gold/2.2.0.5/etc/goldd.conf
and inside the file set:
database.user = gold
database.password = Ch@ngeMe
Start Gold:
# /cm/shared/apps/gold/2.2.0.5/sbin/goldd
** a startup script can be placed under /etc/init.d with the name gold in order to start/stop Gold as a service
Now you are now ready to define users, projects, machines, accounts etc. as necessary for your site.
Define Users:
(On one line:)
# gmkuser -n "Bright Support" -E "support@brightcomputing.com" cmsupport
Define Machines:
# for i in `seq -w 1 100`; do gmkmachine -d "cn$i" cn$i; done
Modify Machines:
# for i in `seq -w 1 100`; do gchmachine -d "cn$i" -m cn$i; done
Remove Machines:
# grmmachine cn001
Define Projects:
# gmkproject -d "Bright Test" cmsupport
Successfully created 1 Project
Auto-generated Account 1
Add Users to the Projects
# gchproject --addUsers cmsupport cmsupport
Successfully created 1 ProjectUser
Make Deposit
Add 360000000 credits to the cmsupport project, to be used from 2015 till 2016:
# gdeposit -p cmsupport -z 360000000 -s 2015-01-01 -e 2016-01-01
# glsalloc
Id Account Active StartTime EndTime Amount CreditLimit Deposited Description
-- ------- ------ ---------- ---------- --------- ----------- --------- --------------
1 1 True -infinity infinity 0 0 0 Auto-Generated
2 1 True 2015-01-01 2016-01-01 360000000 0 360000000 Check Balance:
# gbalance -p cmsupport
Id Name Amount Reserved Balance CreditLimit Available
-- --------- --------- -------- --------- ----------- ---------
1 cmsupport 360000000 0 360000000 0 360000000
# gbalance -p cmsupport -u cmsupport
Id Name Amount Reserved Balance CreditLimit Available
-- --------- --------- -------- --------- ----------- ---------
1 cmsupport 360000000 0 360000000 0 360000000
# gbalance -u cmsupport
Id Name Amount Reserved Balance CreditLimit Available
-- --------- --------- -------- --------- ----------- ---------
1 cmsupport 360000000 0 360000000 0 360000000
# gbalance -u cmsupport -m cn001
Id Name Amount Reserved Balance CreditLimit Available
-- --------- --------- -------- --------- ----------- ---------
1 cmsupport 360000000 0 360000000 0 360000000** Now you are ready to run some jobs. Before doing so you will need to integrate Gold with your Resource Management System. Although the quotation, reservation, and charge steps will most likely be invoked automatically by your resource management system, it is useful to understand their effects by invoking them manually.
Simulate the lifecycle of a job
Assume the job has the following characteristics:
Job Id:
Job Name:
User Name: cmsupport
Project Name: cmsupport
Machine Name: cn001-cn100
Requested Processors: 24
Estimated WallClock: 3600 seconds
Actual WallClock: 1234 seconds
** When a job is submitted, it is useful to check that the user’s account has enough funds to run the job. This will be verified when the job starts, but by that point the job may have waited some time in the queue only to find out it never could have run in the first place. The job quotation step can fulfill this function. Additionally, the quote can be used to determine the cheapest place to run, and to guarantee the current rates will be used when the job is charged.
How much it will cost to run the job
# gquote -p cmsupport -u cmsupport -m cn001 -P 24 -t 3600
Successfully quoted 86400 credits
Make Reservation:
When a job starts, the resource management system creates a reservation (or pending charge) against the appropriate allocations based on the estimated wallclock limit specified for the job.
# greserve -J 130.cn-mgmt -p cmsupport -u cmsupport -m cn001 -P 24 -t 3600
Successfully reserved 86400 credits for job 130.cn-mgmt and created gold job id 1
[root@cn-mgmt ~]# gbalance
Id Name Amount Reserved Balance CreditLimit Available
-- --------- --------- -------- --------- ----------- ---------
1 cmsupport 360000000 86400 359913600 0 359913600Charge Jobs:
After a job completes, any associated reservations are removed and a charge is issued against the appropriate allocations based on the actual wallclock time used by the job.
# gcharge -J 130.cn-mgmt -u cmsupport -p cmsupport -m cn001 -P 24 -X WallDuration=2400
# gbalance -p cmsupport --total
Balance
---------
359942400
The account balance is 359942400 creditsRefund Jobs:
# grefund -J 130.cn-mgmt
Successfully refunded 57600 credits for job 130.cn-mgmt
Check Project Accounting Statement:
# gstatement -p cmsupport
Check Project Usage:
# gusage -p cmsupport
Integrate with WLM
Delayed Accounting
In the absence of a dynamic system, some sites enforce allocations by periodically (weekly or nightly) parsing resource manager job logs and then applying debits against the appropriate project accounts. Although Gold can easily support this type of system by the use of the qcharge command in post-processing scripts, this approach allows users or projects to use resources significantly beyond their designated allocation and generally suffers from stale accounting information.
Dynamic Accounting
Gold’s design allows it to interact dynamically with your resource management system. Charges for resource utilization can be made immediately when the job finishes (or even incrementally throughout the job). Additionally, reservations can be issued at the start of a job to place a hold against the user’s account, thereby ensuring that a job will only start if it has sufficient reserves to complete. The remainder of this document will describe the interactions for dynamic accounting.
Add an appropriate AMCFG line into maui.cfg to tell Maui how to talk to Gold:
# vim /cm/shared/apps/maui/current/spool/maui.cfg
The AMCFG line is added in this kind of format:
AMCFG[bank] TYPE=GOLD HOST=cm-mgmt.cm.cluster PORT=7112 SOCKETPROTOCOL=HTTP WIREPROTOCOL=XML CHARGEPOLICY=DEBITALLWC JOBFAILUREACTION=IGNORE TIMEOUT=15
Add a CLIENTCFG line into maui-private.cfg to specify the shared secret key. This secret key will be the same secret key specified in the “make auth_key” step:
# cat /cm/shared/apps/maui/current/spool/maui-private.cfgCLIENTCFG[AM:bank] KEY=Ch@ngeMe AUTHTYPE=HMAC64
** Gold will need to allow the the user id that Maui runs under to perform scheduler related commands (Job Charge, Reserve, Quote, etc).
Allow the maui user to use Gold:
# gmkuser -d "Maui Scheduler" maui
# goldsh RoleUser Create Role=Scheduler Name=maui
Methods of interacting with Gold
http://docs.adaptivecomputing.com/gold/x3877.php
Interaction Points
http://docs.adaptivecomputing.com/gold/x3841.php
References:
http://docs.adaptivecomputing.com/gold/
Downloads: