Does each jupyter notebook use threads or processes

user_10302452 0
Viewed 4808

When I run the command

jupyter notebook

from the bash shell this starts a jupyter notebook server on localhost. Every time I open a new notebook and start executing code, is that a new process on a separate core or a new thread on the same core? Suppose I were to run N different independent notebooks which are CPU heavy, on a machine with M cores

  1. If they all run from the same core (same process) with different threads then are the rest of the cores are basically idle?
  2. If they run from different cores, then does that mean I can only run M total notebooks at the same time?
  3. Or perhaps its a mix of both: the ipython kernel takes care of launching new threads/processes that are beyond my manual control. In which case only M total notebooks would be able actually execute concurrently at the interpreter level, but I could open as many notebooks as my memory would allow.
jupyter-notebook python
2 Answers

I did an experiment on my laptop which has 2 cores, max 4 threads. I ran between 2-5 identical notebooks which each incremented a number from 1-100 trillion. It took a few minutes for each notebook to finish executing.

In windows task manager, each notebook was considered its own process. No more than 4 notebooks could execute simultaneously, with each one consuming between 20-25% of a single CPU. The 5th notebook effectively had to wait until the others were done before it began executing.

So to answer my own questions

  1. Each execution of a notebook is a separate thread, capable of running on any of the cores of your computer. Cells in the same notebook cannot execute concurrently (obviously) without using a library like Threading.

  2. You can have as many notebooks open as your RAM permits, assuming they don't execute. However, you can only run

    MAX_THREADS = THREADS_PER_CORE*NUM_CORES

    notebooks simultaneously. If you try to run more, the additional notebooks will just stall until the other threads finish.

  3. This is the correct interpretation

user_103024520

From what I can tell from reading the documentation, the bug reports, and using it myself, each jupyter notebook command starts it's own worker process, and that's the extent of what's available for running directly. It also is possible to send long-running jobs to separate worker tasks with an add-on or with various concurrency modules.

Admin1
Related