Scripting with cmsh

Sometimes it’s useful to be able to run basic commands against cmsh in an automated way. We often take the tool for granted but there are a number of subtle tricks that can go a long way to making the use of the command more resilient.

While this can be very helpful for small, simple tasks, for more complicates tasks we recommend investigating the Python API which may be more suitable.

Exit Codes

One important factor when scripting with cmsh is understanding what errors will be visible to a calling application. This exit code can be used to prematurely exit a bash script or other tool before bad data becomes a problem.

In normal operation with cmsh -c or cmsh -f

Exit CodeReason
0cmsh connected successfully AND the last command ran without error
non 0cmsh failed to connect OR the last command failed

The -q flag to cmsh will change it’s behavior so that it will fail on the first command error

Exit CodeReason
0cmsh connected successfully AND ALL commands ran without error
non 0cmsh failed to connect OR ANY command failed

Examples

With -c alone we see that an error in the middle is not exposed to the calling script

# cmsh -c "device; error; list"; echo $?
Command not found: error
Type                   Hostname (key)     MAC                Category         Ip              Network        Status          
---------------------- ------------------ ------------------ ---------------- --------------- -------------- ----------------
HeadNode               ew-b91-c7u9-09-28  FA:16:3E:DA:D4:F2                   10.141.255.254  internalnet    [   UP   ]      
PhysicalNode           node001            FA:16:3E:C5:0A:BD  default          10.141.0.1      internalnet    [   UP   ]      
0

But if the error is the last command cmsh returns non-0.

# cmsh -c "device; list; error"; echo $?
Type                   Hostname (key)     MAC                Category         Ip              Network        Status          
---------------------- ------------------ ------------------ ---------------- --------------- -------------- ----------------
HeadNode               ew-b91-c7u9-09-28  FA:16:3E:DA:D4:F2                   10.141.255.254  internalnet    [   UP   ]      
PhysicalNode           node001            FA:16:3E:C5:0A:BD  default          10.141.0.1      internalnet    [   UP   ]      
Command not found: error
1

And here we can see the difference when we pass -q to cmsh, even when the error is in the middle of the commands it will fail and return an error. The list command is not executed because it falls after cmsh has exited it’s run.

# cmsh -q -c "device; error; list"; echo $?                                                                                                                                                                                       
Command not found: error
1

Other notable command line flags

FlagAction
-f filenameRather than run commands from the command line read them from a file
--tty/-tForce tty mode where headers are placed on listings
--echo/-xEcho all commands

Examples

Running commands from a file

# cat test
device
use node001
get category
# cmsh -q -f test
default

Forcing tty mode so that headers will be present

# cat <(cmsh -q -c 'device; list')
HeadNode               ew-b91-c7u9-09-28  FA:16:3E:DA:D4:F2                   10.141.255.254  internalnet    [   UP   ]      
PhysicalNode           node001            FA:16:3E:C5:0A:BD  default          10.141.0.1      internalnet    [   UP   ]      
# cat <(cmsh --tty -q -c 'device; list')
Type                   Hostname (key)     MAC                Category         Ip              Network        Status          
---------------------- ------------------ ------------------ ---------------- --------------- -------------- ----------------
HeadNode               ew-b91-c7u9-09-28  FA:16:3E:DA:D4:F2                   10.141.255.254  internalnet    [   UP   ]      
PhysicalNode           node001            FA:16:3E:C5:0A:BD  default          10.141.0.1      internalnet    [   UP   ]      

Echo mode to capture the commands that were executed.

# cmsh -q -x -f test
+(Wed Sep 29 22:16:35 2021) device
+(Wed Sep 29 22:16:35 2021) use node001
+(Wed Sep 29 22:16:35 2021) get category
default

Formatting and Delimiters for easier consumption of data

By default all entity values have a default column width and results longer may be cropped. In order to preserve the full width of the value you can use:0 as the format width.

# cat <(cmsh -q -c 'category list -f name,softwareimage')
default              default-image       
this is an entity w+ default-image       
this-is-a-really-lo+ default-image    
# cat <(cmsh -q -c 'category list -f name:0,softwareimage:0')
default                                                                             default-image
this is an entity with spaces which you should not normally do but it is an example default-image
this-is-a-really-long-category-name-that-is-way-longer-than-normal                  default-image

By using :0 we have preserved the full width of the variables. The problem now is that we need some way of delimiting between columns in the case a column may contain spaces. The format command of cmsh does allow for the use of custom delimiters. Below we use a tab and can see the changes by using cat -A.

# cat -A <(cmsh -q -c 'category list -d \t -f name:0,softwareimage:0')
default                                                                            ^Idefault-image$
this is an entity with spaces which you should not normally do but it is an example^Idefault-image$
this-is-a-really-long-category-name-that-is-way-longer-than-normal                 ^Idefault-image$

We could have just as easily used a comma or a semicolon.

# cat -A <(cmsh -q -c 'category list -d , -f name:0,softwareimage:0')
default                                                                            ,default-image$
this is an entity with spaces which you should not normally do but it is an example,default-image$
this-is-a-really-long-category-name-that-is-way-longer-than-normal                 ,default-image$

Outputting data in JSON

Later versions of Bright Cluster Manager also allow the output of data directly to JSON which may be more useful for some applications. This is achieved by using the special format code {}. In this mode, the length specifier has no effect.

# cmsh -c 'device; list -t computenode -d {} -f hostname,ip,category,mac'
[
  {
    "category": "default",
    "hostname (key)": "node001",
    "ip": "10.141.0.1",
    "mac": "FA:16:3E:C4:28:1C"
  }
]

Concurrent Operation

When operating from a script there is the possibility of concurrent operation, for instance, if the script is running as the result of an action. There is the possibility of race conditions when multiple instances of cmsh operate on the same entities, this is because cmsh keeps a cache of the entity, updates it in memory, and then applies it back to the database. If multiple scripts load at the same time and all update the same entity the last one to update the entity may overwrite the updates made by previous attempts.

When scripting, where a concurrent option is possible, scripts should be written in a way to prevent this race condition using one of the readily available tools. Below is an example of using flock. Note this only prevents this one script from running multiple times, other scripts or interactive sessions could cause an issue if they are operating on the same entity.

#!/bin/bash

overlay_from=$1
overlay_to=$2

(
# THIS SCRIPT MAY BE EXECUTED IN PARALLEL AND MUST BE LIMITED TO ONE
# cmsh INSTANCE AT A TIME TO PREVENT OVERWRITING NODE LIST

# Wait up to 30 seconds to get an exclusive lock before running cmsh
flock -x -w 30 200 || exit 1

# move nodes
/cm/local/apps/cmd/bin/cmsh -q -x -c "configurationoverlay; movenodes $overlay_from $overlay_to -a $CMD_ENTITY_NAME;commit;" >> /cm/shared/examples/move_node_overlay.txt

) 200> /var/lock/.move_node_overlays.exclusivelock

The inner subshell opens file descriptor 200 as a file in /var/lock, the flock command will wait up to 30 seconds to get an exclusive lock on the file before executing cmsh.

Updated on September 8, 2023

Related Articles

Leave a Comment