Python include while loop inside parallelized task

user_14649310 0
Viewed 69

I am trying to build a parallelized task where aset of calculations happen in paraller with different set o parameters. Then in each of these independent tasks there are some while loops that for each of these tasks run through a lit of parameters.

The result of each parallelized task is saved in a csv file. However the while loops seem to not progress each parallelized task stops at the first iteration of the while loop.

Any possible explanations for that?

The actual code is way to big to post here but an abstraction could be this:

def inner_func(a,b):
    while a<b:
        write_to_file(a)
        a++

if __name__ == '__main__':
    arguments=[(a,b),(b,c)]
    with multiprocessing.Pool(processes=6) as pool:
        pool.map(inner_func, arguments)
multiprocessing python
2 Answers

Code you provided contains bugs, I have fixed it to:

import multiprocessing


def inner_func(input):
    a = input[0]
    while a < input[1]:
        print(a)
        a = a + 1


if __name__ == '__main__':
    arguments = [(1, 4), (5, 8)]

    with multiprocessing.Pool(processes=6) as pool:
        pool.map(inner_func, arguments)

Executing resulted in:

Result

Does it resolve your issue?

user_76674310

I found the problem it was not within the code. It is just that Jupyter Notebook for some reason when trying to run parallel processes you have to import functions as modules instead of defining them within the same notebook where your main function is.

user_146493100
Related