Is there any way to make the Python garbage collection deterministic?

Viewed 99

So I have the following problem:

I want to measure the memory complexity of a function and I am using the following code. For memory measurement the module memory_profiler is used. Apparently, this module is not subtracting already allocated memory automatically, so I tried to subtract allocated memory. Problem seemingly is, that even when I invoke the garbage collector manually (and mark all relevant variables with del), memory gets freed while I am trying to measure the consumption of the function. This destroys the measurement. Is there any way to work around this problem and prevent the garbage collection from interfering?

for key in datasets.keys():
        
        stats[key] = {"Runtime" : None, "Memory" : None}

        del data
        
        del memory
        
        del start
        
        del end
        
        del initialMem
        
        gc.collect()

        data =  datasets[key]["Train"]["X"].iloc[datasets[key]["Train"]["subset"],:].reset_index(drop=True)

# start of measurement
        
        initialMem = memory_usage(-1)
        
        start = time.time()

        memory = memory_usage((buildDistanceMatrix,(data, distanceMetric)))

        end = time.time()

        stats[key]["Runtime"] = end - start
            
        stats[key]["Memory"] = max(memory) - max(initialMem)
0 Answers
Related