Python reclaiming memory after deleting items in a dictionary

user_129475 0
Viewed 7292

I have a relatively large dictionary in Python and would like to be able to not only delete items from it, but actually reclaim the memory back from these deletions in my program. I am running across a problem whereby although I delete items from the dictionary and even run the garbage collector manually, Python does not appear to be freeing the memory itself.

A simple example of this:

>>> tupdict = {}
# consumes around 2 GB of memory
>>> for i in xrange(12500000):
...   tupdict[i] = (i,i)
... 
# delete over half the entries, no drop in consumed memory
>>> for i in xrange(7500000):
...   del tupdict[i]
... 
>>> import gc
# manually garbage collect, still no drop in consumed memory after this
>>> gc.collect()
0
>>>

I imagine what is happening is that although the entries are deleted and garbage collector run, Python does not go ahead and resize the dictionary. My question is, is there any simple way around this, or am I likely to require a more serious rethink about how I write my program?

memory-management python
2 Answers

You're right that Python doesn't resize dictionary back if items are deleted from dictionary. This have nothing to do with OS memory management and garbage collection, it is an implementation detail of Python's dict data structure.

A workaround is to create a new dictionary by copying the old dictionary. Check this great video for more info: http://pyvideo.org/video/276/the-mighty-dictionary-55 (around 26:30 there is an answer).

user_1147950
Related