Why multiprocessing in Anaconda is slower than in Python

user_10580661 0
Viewed 593

For example, the same code in anaconda environment is running noticeably slower than in bare python env

import multiprocessing as mp
from tqdm.auto import tqdm

def summer(i):
    return i+i

with mp.Pool(6) as pool:
    data = range(100000)
    results = tqdm(pool.imap_unordered(summer, data), total=len(data))
    results = list(results)
    pool.close()
    pool.join()

time in python env:

$time python testscript.py 
100%|████████████████████████████| 100000/100000 [00:02<00:00, 45094.94it/s]

real    0m2,316s
user    0m3,953s
sys     0m1,525s

time in conda env:

$time python testscript.py 
100%|████████████████████████████| 100000/100000 [00:03<00:00, 26315.68it/s]

real    0m3,873s
user    0m6,157s
sys     0m1,912s

What could be the underlying reason of it? Tried profiling the code with viztracer, but it simply shows that summer() takes longer to execute. Follow-up question: Is there a way to profile deeper in python?

anaconda profiling multiprocessing python
1 Answers

It really seems odd that your 3.7 is faster than 3.8

Reasons 3.8 should be faster than 3.7

Likely culprits
  • The default protocol in the pickle module is now Protocol 4 (pickle is used to send args and receive returns from child processes)
  • Reduced an overhead of converting arguments passed to many builtin functions and methods (also probably benefits unpacking args from pickle stream)

Less likely culprits

  • The interpreter loop has been simplified by moving the logic of unrolling the stack of blocks into the compiler.
  • All GC tracked objects (e.g. tuple, list, dict) size is reduced 4 or 8 bytes.

Example of speed changes over several versions of python for various operations (also page for 3.8 changelog):

https://docs.python.org/3/whatsnew/3.8.html#demos-and-tools

user_32201350
Related