Python multiprocessing.Queue maximum size of a single item

Question

I'm offloading a task to a separate process to protect my memory space. The process runs a cythonized C library that tends to not fully clean up after itself. The result is then returned through multiprocessing.Queue. However, once the item being return reaches a certain size, the Queue.get method stalls.

I'm using the processify wrapper from https://gist.github.com/schlamar/2311116 which wraps the function call in a Process.

My test function is

@processify
def do_something(size: int):
    return np.zeros(shape=size, dtype=np.uint8)

My test code is

if __name__ == '__main__':

    for k in range(1, 11):
        print(f"{k*256}MB")
        t0 = time()
        do_something(k * 256 * 1024 * 1024)
        print(f"Took {time()-t0:0.1f}s")

This runs smoothly until 2048MB where it just stays for minutes (no CPU activity) until I cancel the process:

256MB
Took 0.7s
512MB
Took 1.6s
768MB
Took 2.1s
1024MB
Took 2.7s
1280MB
Took 3.4s
1536MB
Took 4.0s
1792MB
Took 4.6s
2048MB
^CTraceback (most recent call last):
  File "processify.py", line 63, in 
    do_something(k * 256 * 1024 * 1024)
  File "processify.py", line 40, in wrapper
    ret, error = q.get()
  File "/home/.../python3.7/multiprocessing/queues.py", line 94, in get
    res = self._recv_bytes()
  File "/home/.../python3.7/multiprocessing/connection.py", line 216, in recv_bytes
    buf = self._recv_bytes(maxlength)
  File "/home/.../python3.7/multiprocessing/connection.py", line 407, in _recv_bytes
    buf = self._recv(4)
  File "/home/.../python3.7/multiprocessing/connection.py", line 379, in _recv
    chunk = read(handle, remaining)
KeyboardInterrupt

From the stack trace it becomes evident that the Queue.get function is waiting. If I add print statements, I can see that Queue.put has already finished at this time so the return value should be inside the Queue. I also tried to run without Process.join as suggested by a commend in the GitHub gist, but that didn't help either.

I know that this kind of design is suboptimal and I should probably fix the cython library so that I don't need to offload in the first place. Yet, I would like to know if there's an inherent limitation in python's multiprocessing that doesn't allow for objects of a certain size to pass through the Queue.

Thank you all in advance!

Python multiprocessing.Queue maximum size of a single item

0 Answers