How to make sure that several processes has started using pool

Question

in the below code, in postTasksForNSegments() method, i am creating a process in a pool for each iteration. to confirm that there are several process started in every iteration i placed the print-statement print(f".current_process:{multiprocessing.current_process}")

At run-time, i received the below posted output for the print-statement. Given the memory address as stated below which is 0x0000017F7A4F1D80, does that mean i did not start several processes and it is only one process started?

please let me know why i am getting the same memory address every time? please tell me how to make sure that i did start several process in a pool?

if the code below indicates that it is only one process started, please let me know how to modify the code to start several processes

code:

@staticmethod
def initPool():
    DebugGridCellsProcessingPerNRowsUsingPool.pool = Pool(cpu_count() -1)

@staticmethod
def closePool():
    DebugGridCellsProcessingPerNRowsUsingPool.pool.close()
    DebugGridCellsProcessingPerNRowsUsingPool.pool.join()   

def postTasksForNSegments(self):
    for i in range (0,self.numOfRows):
        self.res = DebugGridCellsProcessingPerNRowsUsingPool.pool.map_async(self.run,[[
        tasksCarierForRowsOfGridCellsClassifications,
        tasksCarierForRowsOfNDVIsTIFFDetails,
        tasksCarierForRowsOfPixelsValuesSatisfyThresholdInTIFFImageDatasetCnt,
        tasksCarierForRowsOfPixelsValuesDoNotSatisfyThresholdInTIFFImageDatasetCnt,
        tasksCarierForRowsOfpixelsValuesSatisfyThresholdInMixedNZCAndZCCnt,
        ....
        ....
        ....
        tasksCarierForRowsOfPixelsValuesOfNoDataInNoDataCell,
        ]],
        chunksize=self.numOfRows // cpu_count()
        )
        DebugGridCellsProcessingPerNRowsUsingPool.procs.append(self.res)
        print(f".current_process:{multiprocessing.current_process}")

output of print-statement in the above code

numOfRows:30
(numOfSegments:3
.current_process:
numOfRows:27
(numOfSegments:3
.current_process:
numOfRows:24
(numOfSegments:3
.current_process:
numOfRows:21
(numOfSegments:3
.current_process:
numOfRows:18
(numOfSegments:3
.current_process:
numOfRows:15
(numOfSegments:3
.current_process:
numOfRows:12
(numOfSegments:3
.current_process:
numOfRows:9
(numOfSegments:3
.current_process:
numOfRows:6
(numOfSegments:3
.current_process:
numOfRows:3
(numOfSegments:3
.current_process:

userFd75 · Answer

You are printing from the parent process, not from a worker process.

To print from the worker process, you should put the print statement in the run method. To uniqely identify a process, it is better to use os.getpid().

Some style remarks;

Instead of a for-loop with map_async, why not use a simple map (or imap_unordered)?
The length of the variable names in combination with CamelCase makes the code hard to read.

Edit2:

i tested the code on a single process and surprisingly, the processing time using single cpu is must faster the it using multi-cpus. any reason for that please?

Your question doesn't contain enough information to be sure. What follows is my best guess.

There is an amount of overhead associated with multiprocessing. Think about this: All the data that you sumbit to apply_async in args has to be packaged (using pickle) and sent to the worker processes via interprocess communication and unpackaged. The same goes for any data returned by the workers! If the amount of data is large, the time for that needed could overwhelm the time needed for the execution of the run method.

If large amounts of data have to be sent to or received from workers, I would recommend writing that data to a file on an SSD or RAM-disk, and send the workers merely the name of the input file. Similarly the workers should only return the (random!) name of any output file.

Hint: You can use the cProfile module to get an idea about where your programs are spending there time. On my website I've published several articles about profiling Python programs. Here is the start of that series. If you want to improve program performance, you should always start with profiling.

How to make sure that several processes has started using pool

1 Answers