Storage and Python environments

What files are being saved in my Jupyter environment?

• Files under /home/<your_username> only,
• Any conda environments you may create.

​

Any other library or file will be discarded after your session is terminated.

Filesystem

Once your Jupyter session starts, several filesystems are mounted in different locations.

• A read-only filesystem that contains code examples under /home/<your_username>/public,
• A filesystem that contains your persistent files under /home/<your_username>, that is shared across all Jupyter instances you may have,
• All other locations are not persisted but much faster. As the default user is jovyan, you may use /home/jovyan and /tmp/ for your IO-intensive operations.

​

Note: The filesystems under /home/<your_username> for archiving, as it is linked to a S3 File System which is much slower than a regular filesystem.

All io-intensive operations (such as compressing/uncompressing) will run much faster if you use another directory.

Python Libraries

Some libraries like Tensorflow uses the imp module to import some dependencies dynamically. This triggers the execution of find commands, that are known to be slow on a S3 Filesystem.

If you notice that a library takes a long time to be imported, use the following snippet to import it without triggering a find operation on your S3 FS.

## BEGIN S3FS IMPORT SNIPPET ##
import os, sys
s3_home = os.path.join('/home', os.environ.get('JUPYTERHUB_USER'))
try: sys.path.remove(s3_home)
except Exception: pass
os.chdir('/home/jovyan')

# BEGIN PYTHON IMPORTS #
import numpy
import scipy
# END PYTHON IMPORTS #

os.chdir(s3_home)
sys.path.append(s3_home)
## END S3FS IMPORT SNIPPET ##

Conda environments

Your Jupyter instance is initialized with default conda environments, that are reset after each session restart.

List the current environments

• Open a shell terminal to type the following commands, or type them directly in a notebook using the ! character.

(base) jovyan@jupyter-<your-username>:/home/<your-username>$ conda env list
# conda environments:
#
base               *     /opt/conda
climaf                   /opt/conda/envs/climaf
pcmdi                    /opt/conda/envs/pcmdi

These three default environments are installed by default and are not persisted across sessions.

However, any custom environment you may create will be persisted.

Create a custom environment

• Execute these commands with your own <env-name>, from a terminal or a notebook using the ! character.

(base) jovyan@jupyter-<your-username>:/home/<your-username>$ conda create -n <env-name> ipykernel
(base) jovyan@jupyter-<your-username>:/home/<your-username>$ conda activate <env-name>
(<env-name>) jovyan@jupyter-<your-username>:/home/<your-username>$ python -m ipykernel install --user --name <env-name>

• Finally, refresh the page using your web browser.

You should now be able to pick your newly created environment in the JupyterHub launcher.

Opening a launcher tab, you will need to select between two kernels in order to create a new empty Jupyter notebook.