I have a super huge numpy array which memory allocated to it never gets free again. I simply demonstrate my situation so you can see the problem yourself.
Memory allocated to simple numpy arrays will immediately freed up after that variable can be remove (like below which I delete it):
import numpy as np
X = np.ones((40000, 40000))
X.nbytes
12800000000
del(X)
When I run the code above, all the 12 GB memory will free up immediately. But in case of nested numpy arrays things get complicated:
import numpy as np
import random
foo = np.array([np.array([np.ones((256,)) for j in range(random.randint(100, 150))]) for i in range(40000)])
sum(f.nbytes for f in foo)
10240481280
del(foo)
Now the 10 GB of memory will never gets freed even if you run gc.collect() explicitly. Do you guys have any clue?
P.S: The env: Ubuntu + Python 2.7 + numpy 1.15.1