If you run the “sshare” tool for Slurm 23.02 on your cluster, you may see the following error output:
$ sshare
sshare: error: plugin_load_from_file: dlopen(/cm/shared/apps/slurm/23.02.2/lib64/slurm/priority_basic.so): /cm/shared/apps/slurm/23.02.2/lib64/slurm/priority_basic.so: undefined symbol: job_list
sshare: error: Couldn’t load specified plugin name for priority/basic: Dlopen of plugin file failed
sshare: error: cannot create priority context for priority/basic
sshare: fatal: failed to initialize priority plugin
This issue is caused by an upstream bug:
https://bugs.schedmd.com/show_bug.cgi?id=16822
These upcoming releases for Bright versions that already provide packages for Slurm 23.02 will provide packages for Slurm 23.02.4, which will contain the fix for this issue:
- 9.2-14
- 9.1-18
- 9.0-21
Until those packages are available, uncommenting the following line in the slurm.conf file will provide a workaround:
#PriorityType=priority/multifactor
Uncommenting the above line will have Slurm use priority/multifactor instead of priority/basic, which is the priority type that is affected by the upstream bug.
In order to apply the change to PriorityType, you will need to restart slurmctld on all head nodes. You may do so by running the following cmsh command on the cluster’s active head node:
$ cmsh -c “device foreach -t headnode (services; restart slurmctld)”