So I have the following problem:
I want to measure the memory complexity of a function and I am using the following code. For memory measurement the module memory_profiler is used. Apparently, this module is not subtracting already allocated memory automatically, so I tried to subtract allocated memory. Problem seemingly is, that even when I invoke the garbage collector manually (and mark all relevant variables with del), memory gets freed while I am trying to measure the consumption of the function. This destroys the measurement. Is there any way to work around this problem and prevent the garbage collection from interfering?
for key in datasets.keys():
stats[key] = {"Runtime" : None, "Memory" : None}
del data
del memory
del start
del end
del initialMem
gc.collect()
data = datasets[key]["Train"]["X"].iloc[datasets[key]["Train"]["subset"],:].reset_index(drop=True)
# start of measurement
initialMem = memory_usage(-1)
start = time.time()
memory = memory_usage((buildDistanceMatrix,(data, distanceMetric)))
end = time.time()
stats[key]["Runtime"] = end - start
stats[key]["Memory"] = max(memory) - max(initialMem)