I am looking for more insights on the Queues implementations in Python than I can find in the documentation.
From what I understood, and excuse my ignorance if I am wrong on this:
queue.Queue(): is implemented through basic arrays in-memory and so cannot be shared between multiple processes but can be shared between threads. So far, so good.
multiprocessing.Queue(): is implemented through pipes (man 2 pipes) which have size limit (rather tiny: on Linux, man 7 pipe says 65536 untweaked):
Since Linux 2.6.35, the default pipe capacity is 65536 bytes, but the capacity can be queried and set using the
fcntl(2)F_GETPIPE_SZandF_SETPIPE_SZoperations
But, in Python, whenever I try to write data larger than 65536 bytes into the pipe, it works without exception - I could flood my memory this way:
import multiprocessing
from time import sleep
def big():
result = ""
for i in range(1,70000):
result += ","+str(i)
return result # 408888 bytes string
def writequeue(q):
while True:
q.put(big())
sleep(0.1)
if __name__ == '__main__':
q = multiprocessing.Queue()
p = multiprocessing.Process(target=writequeue, args=(q,))
p.start()
while True:
sleep(1) # No pipe consumption, we just want to flood the pipe
So here are my questions:
does Python tweak the pipe limit? if yes, by how much ? Python source code is welcomed.
Are Python piped communications inter-operable with other non-Python processes? If yes, working examples (JS preferably) and resource links are welcomed.