Bright Cluster Manager’s data science add-on provides many ML-related packages that can be used to run AI workloads on a Bright cluster without having to use container images. In addition, it is also possible to run AI workloads by using container images from e.g. the NVIDIA GPU Cloud.
With Bright’s Jupyter integration, data scientists and ML practitioners can conveniently run applications by writing notebooks in their browsers. This workflow makes it easy to run different experiments, enables rapid prototyping, and allows fine-tuning ML models.
However, writing Jupyter notebooks in a browser may prove to be limiting or complex while developing large applications. Scientists may want to take advantage of a rich set of features provided by a Python integrated development environment (IDE), such as code inspection, autocompletion, and on-the-fly error checking.
This article will describe how to run Jupyter kernels in this advanced scenario, thanks to two of the most popular tools for Python development: PyCharm Professional and Visual Studio Code.
Both PyCharm Professional and Visual Studio Code support Jupyter Notebook.
PyCharm Professional natively supports Jupyter notebook servers, while Visual Studio Code supports Jupyter Notebook via its Jupyter extension.
The steps required to configure a remote Jupyter server are described in a dedicated section of the IDEs documentation, both for PyCharm and for Visual Studio Code.
These guides should be well suited to connect PyCharm or Visual Studio Code to a Bright Cluster where Jupyter is deployed. Unfortunately, this does not work as expected out of the box.
Presently developers can’t set up a remote server directly pointing to the Jupyter cluster because both PyCharm Professional and Visual Studio Code do not support JupyterHub, a fundamental component of Bright’s Jupyter integration. (31 August 2021)
To overcome this limitation, Bright has developed a new tool, the Jupyter Dev Server.
This lightweight component can be executed on the machine where the IDE is installed and will act as a proxy: while running, it will transparently and securely redirect requests from the IDE to the Bright’s JupyterHub cluster.
In the GitHub repository of the tool, you will find further technical details about IDEs’ limitations and helpful notes to install Bright’s proxy.
A walkthrough demo for PyCharm Professional can be watched on this YouTube video. It shows how to set up the dev server and PyCharm Professional, generate tokens for JupyterHub, and run some notebooks with Bright’s kernels.
The procedure to configure Visual Studio Code is similar.
Bright’s Jupyter integration makes the complexity of dealing with cluster resources and workload managers transparent to the end-users. They can take advantage of the Jupyter Kernel Creator Extension to create kernels from their browsers in a simple way.
Bright’s Jupyter Dev Server takes another step in this direction, eliminating the necessity for users to leave their development environment. In fact, while coding in their IDE, the entire Bright cluster resources will be only displayed in the form of a dropdown menu with the available Jupyter kernels.
Developers will not need to know a cluster exists and how it is configured: they will only straightforwardly use its resources while taking advantage of the IDE features and plugins.
