Sequentially running jupyter notebooks with different conda environments

cb_coder 0
Viewed 27

I've got 10 jupyter notebooks, each with many unique package dependencies (that conflict), so I've created a different anaconda environment for each notebook. Each notebook relies on the output of the previous one, which I store and read from local csv files.

Right now I am running each jupyter notebook manually (with their own anaconda environment) to get the final result. Is there a way to run a single script that runs the code of all the jupyter notebooks sequentially (with the correct anaconda environment for each one)?

anaconda jupyter-notebook
1 Answers

You could do it in python and use runipy. You just have to install it with:

pip install runipy

An example on how to use it from the docs:

from runipy.notebook_runner import NotebookRunner
from IPython.nbformat.current import read

notebook = read(open("MyNotebook.ipynb"), 'json')
r = NotebookRunner(notebook)
r.run_notebook()

If you want to run each notebook in a different environment, you can activate each conda environmentfrom a python script. There are multiple ways to do so, one of them is this:

subprocess.run('source activate environment-name && "enter command here" && source deactivate', shell=True)

Replace the "enter command here" with the command you want to run. You don't need the "source deactivate" at the end of the command but it's included just to be safe.

This will temporarily activate the Anaconda environment for the duration of the subprocess call, after which the environment will revert back to your original environment. This is useful for running any commands you want in a temporary environment.

ClaudiaR0
Related