Implementation of multiprocessing.Queue and queue.Queue

Viewed 3353

I am looking for more insights on the Queues implementations in Python than I can find in the documentation.

From what I understood, and excuse my ignorance if I am wrong on this:

queue.Queue(): is implemented through basic arrays in-memory and so cannot be shared between multiple processes but can be shared between threads. So far, so good.

multiprocessing.Queue(): is implemented through pipes (man 2 pipes) which have size limit (rather tiny: on Linux, man 7 pipe says 65536 untweaked):

Since Linux 2.6.35, the default pipe capacity is 65536 bytes, but the capacity can be queried and set using the fcntl(2) F_GETPIPE_SZ and F_SETPIPE_SZ operations

But, in Python, whenever I try to write data larger than 65536 bytes into the pipe, it works without exception - I could flood my memory this way:

import multiprocessing
from time import sleep

def big():
    result = ""
    for i in range(1,70000):
        result += ","+str(i)
    return result # 408888 bytes string

def writequeue(q):
    while True:
        q.put(big())
        sleep(0.1)

if __name__ == '__main__':
    q = multiprocessing.Queue()
    p = multiprocessing.Process(target=writequeue, args=(q,))
    p.start()
    while True:
        sleep(1) # No pipe consumption, we just want to flood the pipe

So here are my questions:

  • does Python tweak the pipe limit? if yes, by how much ? Python source code is welcomed.

  • Are Python piped communications inter-operable with other non-Python processes? If yes, working examples (JS preferably) and resource links are welcomed.

1 Answers
Related