Multiprocessing strange behaviour?

user_7612031 0
Viewed 80

I am using multiprocessing from Python for testing purpose and there are somethings I don't understand.

A priori, processes have their own memory space so we can't share and Python class between them.

But look at my code:

import sys, 
import time
from multiprocessing import Queue, Process

class MainClass():
    def __init__(self, q):
        self.q = q 
        print("Queue in Main", q)

    def start_p(self):
        p = Proc(self.q)
        p.processing()

    def run_p(self):

        p = Process(target=self.start_p, args=())
        p.start()
        return p

class Proc():
    def __init__(self, q):
        self.q = q  

    def processing(self):
        print("Queue in process", self.q)

        n = ''
        try:
            n = self.q.get(0) # Get first item in the "queue"
        except KeyError as e:
            print("NOK", e)

        print("GET: ", n)
        print('Size: ',  self.q.qsize())


if __name__ == "__main__":

    # Creating a queue
    q = Queue()

    # Add 10 numbers in the queue
    for i in range(0,10):
        q.put(i)

    # Add the queue to Main Class
    s = MainClass(q)
    print("Initial queue size", q.qsize())

    # Starting 3 process
    p1 = s.run_p()
    p2 = s.run_p()
    p3 = s.run_p()


    #time.sleep(2)
    print("Final queue size", q.qsize())

I have created a queue on main process, with 10 numbers. Then, I ran 3 process so each one run a task consting of just getting (and delete) first item in a queue.

What I misunderstand is how can this program work and return a final queue 7 ? It seems the queue is shared...but the object itself (multiprocessing) is located in different memory place... But there is no "pointer" mecanism in python ?

The result when I run the programm below:

Result

Behaviour is almost the same on linux except the memory adress is the same for all instances.

l

Please can someone explain me ?

multiprocessing python
1 Answers

Your basic premise is not entirely correct. While multiple processes indeed have their own memory space, that does not mean they cannot exchange data.

In fact there are multiple mechanisms for programs to share data. These are generally called "inter process communication" or IPC. For example;

  • shared memory
  • pipes
  • sockets

Under the covers, a multiprocessing.Queue uses a multiprocessing.Pipe. This in turn is a Python wrapper for an operating system communication pipe. On ms-windows this is an operating system primitive called a named pipe. On many other operating systems pipes are based on file descriptors.

The point is that even when the Queue is inherited in the child processes, they are all connected to the same operating system object. That is basically why they can communicate with each other.

(Note that this is of necessity a simplification; If you really want to know how it works, read the multiprocessing module Python code.)

user_12192950
Related