Can't use Python multiprocessing with large amount of calculations

To reduce memory usage, I suggest you use instead the multiprocessing module and specifically the imap method method (or imap_unordered method). Unlike the map method of either multiprocessing.Pool or concurrent.futures.ProcessPoolExecutor, the iterable argument is processed lazily. What this means is that if you use a generator function or generator expression for the iterable argument, you do not need to create the complete list of arguments in memory; as a processor in the pool become free and ready to execute more tasks, the generator will be called upon to generate the next argument for the imap call.

By default a chunksize value of 1 is used, which can be inefficient for a large iterable size. When using map and the default value of None for the chunksize argument, the pool will look at the length of the iterable first converting it to a list if necessary and then compute what it deems to be an efficient chunksize based on that length and the size of the pool. When using imap or imap_unordered, converting the iterable to a list would defeat the whole purpose of using that method. But if you know what that size would be (more or less) if the iterable were converted to a list, then there is no reason not to apply the same chunksize calculation the map method would have, and that is what is done below.

The following benchmarks perform the same processing first as a single process and then using multiprocessing using imap where each invocation of do_something on my desktop takes approximately .5 seconds. do_something now has been modified to just process a single i, k, j tuple as there is no longer any need to break up anything into smaller lists:

from multiprocessing import Pool, cpu_count
import time

def half_second():
    HALF_SECOND_ITERATIONS = 10_000_000
    sum = 0
    for _ in range(HALF_SECOND_ITERATIONS):
        sum += 1
    return sum

def do_something(tpl):
    # in real case this function takes about 0.5 seconds to finish for each  iteration
    half_second() # on my desktop
    return tpl[0]**2, tpl[1]**2, tpl[2]**2

"""
def generate_tpls():
    for i in range(1, 20-2):
        for k in range(i+1, 20-1):
            for j in range(k+1, 20):
                yield i, k, j
"""

# Use smaller number of tuples so we finish in a reasonable amount of time:
def generate_tpls():
    # 64 tuples:
    for i in range(1, 5):
        for k in range(1, 5):
            for j in range(1, 5):
                yield i, k, j

def benchmark1():
    """ single processing """
    t = time.time()
    for tpl in generate_tpls():
        result = do_something(tpl)
    print('benchmark1 time:', time.time() - t)

def compute_chunksize(iterable_size, pool_size):
    """ This is more-or-less the function used by the Pool.map method """
    chunksize, remainder = divmod(iterable_size, 4 * pool_size)
    if remainder:
        chunksize += 1
    return chunksize

def benchmark2():
    """ multiprocssing """
    t = time.time()
    pool_size = cpu_count() # 8 logical cores (4 physical cores)
    N_TUPLES = 64 # number of tuples that will be generated
    pool = Pool(pool_size)
    chunksize = compute_chunksize(N_TUPLES, pool_size)
    for result in pool.imap(do_something, generate_tpls(), chunksize=chunksize):
        pass
    print('benchmark2 time:', time.time() - t)


if __name__ == '__main__':
    benchmark1()
    benchmark2()

Prints:

benchmark1 time: 32.261038303375244
benchmark2 time: 8.174998044967651

The nested For loops creating the array before the main definition appears to be the problem. Moving that part to underneath the main definition clears up any memory problems.

def chunkIt(seq, num):
    avg = len(seq) / float(num)
    out = []
    last = 0.0

    while last < len(seq):
        out.append(seq[int(last):int(last + avg)])
        last += avg

    return out
 
def do_something(List):
    # in real case this function takes about 0.5 seconds to finish for each 
    iteration
    turn = []
    for e in List:
        turn.append((e[0]**2, e[1]**2,e[2]**2))
    return turn
    

if __name__ == '__main__':
   t1 = time.time()

   List = []
   #in the real case these 20's can go as high as 150
   for i in range(1,20-2):
     for k in range(i+1,20-1):
        for j in range(k+1,20):
            List.append((i,k,j))
            
   t3 = time.time() 
   test = []    

   List = chunkIt(List,3)
   with concurrent.futures.ProcessPoolExecutor() as executor:
        results = executor.map(do_something,List)
        for result in results:
            test.append(result)       
    
   test= np.array(test)
   t2 = time.time()

   T = t2-t1
   T2 = t3-t1