Why does my loop require more memory on each iteration?

Viewed 3336

I am trying to reduce the memory requirements of my python 3 code. Right now each iteration of the for loop requires more memory than the last one.

I wrote a small piece of code that has the same behaviour as my project:

import numpy as np
from multiprocessing import Pool
from itertools import repeat


def simulation(steps, y):  # the function that starts the parallel execution of f()
    pool = Pool(processes=8, maxtasksperchild=int(steps/8))
    results = pool.starmap(f, zip(range(steps), repeat(y)), chunksize=int(steps/8))
    pool.close()
    return results


def f(steps, y):  # steps is used as a counter. My code doesn't need it.
        a, b = np.random.random(2)
        return y*a, y*b

def main():
    steps = 2**20  # amount of times a random sample is taken
    y = np.ones(5)  # dummy variable to show that the next iteration of the code depends on the previous one
    total_results = np.zeros((0,2))
    for i in range(5):
        results = simulation(steps, y[i-1])
        y[i] = results[0][0]
        total_results = np.vstack((total_results, results))

    print(total_results, y)

if __name__ == "__main__":
    main()

For each iteration of the for loop the threads in simulation() each have a memory usage equal to the total memory used by my code.

Does Python clone my entire environment each time the parallel processes are run, including the variables not required by f()? How can I prevent this behaviour?

Ideally I would want my code to only copy the memory it requires to execute f() while I can save the results in memory.

2 Answers
Related