Build Singularity containers to run single user notebook applications
Follow the instructions at https://github.com/zonca/singularity-comet to build images from the
centos_anaconda_jupyterhub.def definition files, or use the containers I have already built on Comet:
These containers have Centos 7 and Ubuntu 16.04 base images, MPI support (not needed for this), Anaconda 4.4.0, the Jupyterhub (for the
jupyterhub-singleuser script) and Jupyterlab (for the awesomeness) packages.
Initial setup of Jupyterhub with Ansible
First we want to use the Ansible playbook provided by the Jupyter team to setup a Ubuntu Virtual Machine, for example on SDSC Cloud or XSEDE Jetstream. This sets up already a Jupyterhub instance on a single machine with Github authentication, NGINX with letsencrypt SSL and spawning of Notebooks as local processes.
Start from: Automated deployment of Jupyterhub with Ansible
It looks like there is a compatibility error with
conda 4.3 and above, I had to fix this (and provided PR upstream), I used the version at https://github.com/zonca/jupyterhub-deploy-teaching/tree/globus_singularity.
In particular check the example configuration file in the
Once we have executed the scripts, connect to the Virtual Machine, login with Github and check that Notebooks are working.
Setup Authentication with Globus
Next we can SSH into the Jupyterhub Virtual Machine and customize Jupyterhub configuration in
oauthenticator should alrady be installed,, but it needs the Globus SDK to support authentication with Globus:
sudo /opt/conda/bin/pip install globus_sdk[jwt]
Then follow the instructions to setup Globus Auth: https://github.com/jupyterhub/oauthenticator#globus-setup
you should now have add these lines in
from oauthenticator.globus import GlobusOAuthenticator c.JupyterHub.authenticator_class = GlobusOAuthenticator c.GlobusOAuthenticator.oauth_callback_url = 'https://xxx-xxx-xxx-xxx.compute.cloud.sdsc.edu/hub/oauth_callback' c.GlobusOAuthenticator.client_id = '' c.GlobusOAuthenticator.client_secret = ''
You should now be able to login with your Globus ID credentials, see the documentation to support credentials from institutions supported by Globus Auth. After login, don’t worry if you get an error in starting your notebook.
Setup Spawning with Batchspawner
In my last post about spawning Notebooks on Comet I was using XSEDE authentication so that each user would have to use their own Comet account.
In this scenario instead we imagine a Gateway system where the administrator shares their own allocation with the Gateway users.
Therefore you should create a SSH keypair for the
root user on the Jupyterhub Virtual Machine and make sure you can login with no need for a password to Comet as the Gateway user.
Then you need to install
git clone https://github.com/jupyterhub/batchspawner.git cd batchspawner/ sudo /opt/conda/bin/pip install .
Then configure the Spawner, see my configuration of Jupyterhub:
You should modify
comet_spawner.py to point to your Gateway user home folder and then fill all the details in
jupyterhub_config.py marked by the
CometSpawner I also create a form for the user to choose the parameters of the job and also the Singularity image they want to use.
Here the spawner uses
SSH to connect to the Comet login node and submit jobs as the Gateway user.
At this point you should be able to login and launch a job on Comet, execute
squeue on Comet to check if that works or look in the home folder of the Gateway user for the logfile of the job and in
/var/log/jupyterhub on the Virtual machine for errors.
Finally we need a way for the gateway Virtual Machine to access the port on the Comet computing node in order to proxy the Notebook application back to the user.
The simpler solution is to create a user
tunnelbot on the VM with no shell access, then create a SSH keypair and paste the private key into the
jupyterhub_config.py file (contact me if you have a btter solution!).
The job on Comet sets up then a SSH tunnel between the Comet computing node and the Jupyterhub VM.
To keep the setup simple, all users are running on the home folder of the Gateway user, for a real deployment, it is possible to create a subfolder for each user beforehand and then use Singularity to mount that as the home folder.