multiprocessing returns "too many open files" but using `with...as` fixes it. Why?

user_316232 0
Viewed 24891

I was using this answer in order to run parallel commands with multiprocessing in Python on a Linux box.

My code did something like:

import multiprocessing
import logging

def cycle(offset):
    # Do stuff

def run():
    for nprocess in process_per_cycle:
        logger.info("Start cycle with %d processes", nprocess)
        offsets = list(range(nprocess))
        pool = multiprocessing.Pool(nprocess)
        pool.map(cycle, offsets)

But I was getting this error: OSError: [Errno 24] Too many open files
So, the code was opening too many file descriptor, i.e.: it was starting too many processes and not terminating them.

I fixed it replacing the last two lines with these lines:

    with multiprocessing.Pool(nprocess) as pool:
        pool.map(cycle, offsets)

But I do not know exactly why those lines fixed it.

What is happening underneath of that with?

process-pool with-statement multiprocessing python
4 Answers

I was already terminating and closing pool but there was a limit on number of file descriptors and I changed my ulimit to 4096 from 1024 and it worked. Following is the procedure:

Check:

ulimit -n

I updated it to 4096 and it worked.

ulimit -n 4096
user_76575970

This can happen when you use numpy.load too, make sure close those files too, or avoid using it and use pickle or torch.save torch.load etc.

user_89299750
Related