Bright Cluster Manager’s data science add-on provides many ML related packages that can be used to run AI workloads on a Bright cluster without having to use container images. In addition, it is also possible to run AI workloads by using container images from e.g. the NVIDIA GPU Cloud. A third option to run AI workload on a Bright cluster is by using packages from Conda. In this article we will show how to use deploy Conda packages on a Bright cluster, and how to use them from Bright’s Jupyter setup.
Bright Jupyter setup supports not only pre-installed kernels, but also allows custom kernels in Conda environments.
We will use Miniconda. If it is not yet installed, then Miniconda can be installed from the official website (https://docs.conda.io/en/latest/miniconda.html). Here we create a sample environment first. On an already-existing setup these steps can be skipped:
wget https://repo.anaconda.com/miniconda/Miniconda3-py39_4.9.2-Linux-x86_64.sh
chmod +x Miniconda3-py39_4.9.2-Linux-x86_64.sh
./Miniconda3-py39_4.9.2-Linux-x86_64.sh
source <install_directory>miniconda3/bin/activate
conda create --name py3.9 python=3.9The kernel template can be installed cluster-wide, or only for particular user(s). The template can be downloaded here (https://support.brightcomputing.com/kb-articles/jupyter-conda/jupyter-eg-kernel-slurm-py37-conda.tar.gz).
If the template is to be made available for all users:
mkdir /cm/shared/apps/jupyter/current/share/jupyter/kerneltemplates
cd /cm/shared/apps/jupyter/current/share/jupyter/kerneltemplates
wget https://support.brightcomputing.com/kb-articles/jupyter-conda/jupyter-eg-kernel-slurm-py37-conda.tar.gz
tar -xzf jupyter-eg-kernel-slurm-py37-conda.tar.gzIf the template is to be made available for a single user:
mkdir ~/.local/share/jupyter/kerneltemplates/
cd ~/.local/share/jupyter/kerneltemplates/
wget https://support.brightcomputing.com/kb-articles/jupyter-conda/jupyter-eg-kernel-slurm-py37-conda.tar.gz
tar -xzf jupyter-eg-kernel-slurm-py37-conda.tar.gzThe Conda environment needs to be activated, and a pip-wheel then needs to be installed. The code in this wheel runs the Jupyter kernel and knows how to communicate with Jupyter Enterprise Gateway:
conda activate py3.9
pip install cm-jupyter-eg-kernel-wlm==2.0.0The Conda kernel template will be available in a Bright tab within the JupyterLab interface, next to the other templates.
Users may now instantiate this template to add new kernels to their Jupyter launcher.