Python Multiprocessing Freeing up memory used by shared array

Viewed 334

Is there a way to deallocate the memory used by a shared Multiprocessing Array ? Here is a code snippet that creates a memory block of approx 1 GiB of shared memory which is then blocked for the remainder of the script. Is there a way to free up that memory during the same script run ?

I tried deleting all references and calling the garbage collector via gc.collect() but that seems to have no effect whatsoever on the Memory usage of the script during runtime.

import psutil
import gc
import ctypes as cty
import multiprocessing as mpc
import os

def bytes_to_GiB_MiB(n_bytes):
    return n_bytes/2**30, (n_bytes%(2**30))/2**20
    
def print_mem_usage():
    process = psutil.Process(os.getpid()) 
    print "Script Mem Usage {:} GiB {:} MiB".format(*bytes_to_GiB_MiB(process.memory_info().rss))
    return process.memory_info().rss    
    
if __name__=="__main__":
    t_N = 5000
    N=150
    Shape_spl = (t_N+4, N,N) 
    mem0 = print_mem_usage()
    C_shared = mpc.Array(cty.c_double, N*N*(t_N+4))
    mem1 = print_mem_usage()
    print "Mem Difference is {:} GiB {:} MiB".format(*bytes_to_GiB_MiB(mem1-mem0))
    del C_shared
    gc.collect()
    # Do something here such that the memory used by C_shared is free again.
    mem2 = print_mem_usage()
    print "Mem Difference is {:} GiB {:} MiB".format(*bytes_to_GiB_MiB(mem2-mem0))
1 Answers

... for the record... This seems to be related to this bug in python <= 3.7
(as already mentioned, it is supposed to be fixed in python 3.8):
https://bugs.python.org/issue32759

There's also a crude "fix" suggested in the discussion which seems to work for me with python 3.7:
(not sure if this also works in python 2.7)

"... delete the globe _heap and recreate a new one"
mp.heap.BufferWrapper._heap = mp.heap.Heap()

Related