Multiprocessing Pool is running the whole code for every process created instead of just the function passed to it

Viewed 1117

The python code does a data pull and pre-processing (to convert the data into an iterable for Pool Map to run) and then calls Pool Map to automatically subset the data and run in parallel. However, I find that the data pull and preprocessing are running in each process again in addition to the function that should be running in each process.

Code is as follows:

#Data Pull
orig_data= pd.read_csv(<fileath>)

#Data Preprocessing
start=time.time()
transformed_input=list() # transformed_input is populated with data from orig_data dataframe as required
end=time.time()
print("Preprocessing Time")
print(end-start)


def examplefunction(transformed_input_sub):
    # Required Function
    return output

if __name__ == '__main__':
    from multiprocessing import Pool
    p=Pool()
    s=time.time()
    output=p.map(examplefunction,transformed_input)
    p.close()
    p.join()
    e=time.time()
    print("Time Taken")  
    print(e-s)

Expected Output:
Preprocessing Time
5.01
Time Taken
10

Actual Output when running on 4 processes:
Preprocessing Time
5.01
Preprocessing Time
5.12
Preprocessing Time
5.35
Preprocessing Time
5.41
Time Taken
10

However, the output of this code is correct (and verified) despite the preprocessing running for each process, which is shouldn't be doing as far as I am aware. The problem then arises when instead of a pulling data from a csv containing only a small portion of the data, I use a Teradata Pull for close to 50 million records. The TD pull without multiprocessing works perfectly fine, but with multiprocessing throws the following error:

No more spool space in database name

This is probably each process is running the TD Pull.
How can I ensure the Pool runs only the function passed to it in parallel and not the complete code from scratch?

I have tried changing the order of code blocks, calling import statements outside the name == 'main' block, and also put the data pull and preprocessing code in a separate python code and called it using exec() within the main program.

0 Answers
Related