1. Home
  2. Machine Learning
  3. Running Jupyter kernel with Conda (Anaconda/Miniconda) environments
  1. Home
  2. Third Party Software
  3. Running Jupyter kernel with Conda (Anaconda/Miniconda) environments

Running Jupyter kernel with Conda (Anaconda/Miniconda) environments

Bright Cluster Manager’s data science add-on provides many ML related packages that can be used to run AI workloads on a Bright cluster without having to use container images. In addition, it is also possible to run AI workloads by using container images from e.g. the NVIDIA GPU Cloud. A third option to run AI workload on a Bright cluster is by using packages from Conda. In this article we will show how to use deploy Conda packages on a Bright cluster, and how to use them from Bright’s Jupyter setup.

Bright Jupyter setup supports not only pre-installed kernels, but also kernels in Conda environments.

We will use Miniconda. If it is not yet installed, then Miniconda can be installed from the official website (https://docs.conda.io/en/latest/miniconda.html). Here we create a sample environment first. On an already-existing setup these steps can be skipped:

wget https://repo.anaconda.com/miniconda/Miniconda3-py39_4.9.2-Linux-x86_64.sh
chmod +x Miniconda3-py39_4.9.2-Linux-x86_64.sh
./Miniconda3-py39_4.9.2-Linux-x86_64.sh
conda create --name py3.9 python=3.9

The kernel template can be installed cluster-wide, or only for particular user(s). The template can be downloaded here (https://support.brightcomputing.com/kb-articles/jupyter-conda/jupyter-eg-kernel-slurm-py37-conda.tar.gz).

If the template is to be made available for all users:

wget https://support.brightcomputing.com/kb-articles/jupyter-conda/jupyter-eg-kernel-slurm-py37-conda.tar.gz
mkdir /cm/shared/apps/jupyter/current/share/jupyter/kerneltemplates
cd /cm/shared/apps/jupyter/current/share/jupyter/kerneltemplates
tar -xzf jupyter-eg-kernel-slurm-py37-conda.tar.gz

If the template is to be made available for a single user:

wget https://support.brightcomputing.com/kb-articles/jupyter-conda/jupyter-eg-kernel-slurm-py37-conda.tar.gz
mkdir ~/.local/share/jupyter/kerneltemplates/
cd ~/.local/share/jupyter/kerneltemplates/
tar -xzf jupyter-eg-kernel-slurm-py37-conda.tar.gz

The Conda environment needs to be activated, and a pip-wheel then needs to be installed. The code in this wheel runs the Jupyter kernel and knows how to communicate with Jupyter Enterprise Gateway:

conda activate py3.9
pip install cm-jupyter-eg-kernel-wlm==2.0.0

The Conda kernel template will be available in a Bright tab within the JupyterLab interface, next to the other templates.

Users may now instantiate this template to add new kernels to their Jupyter launcher.

Updated on April 8, 2021

Was this article helpful?

Related Articles

Leave a Comment