Python: how to get size of all objects in current namespace?

user_5711995 0
Viewed 6405

I have some code that I am running from my own package and the program is using a lot more memory (60GB) than it should be. How can I print the size of all objects (in bytes) in the current namespace in order to attempt to work out where this memory is being used?

I attempted something like 

from pympler import asizeof

for objname in dir():
    print(asizeof.asizeof(thing)/1024) # print size in kb

But it doesn't work as it just prints the size of the string containing the name of the object in the namespace. Is there a way to get an object reference to everything in the namespace in order to use this method or is there a better method for working out what is using the memory?

memory-leaks introspection memory-management size python
3 Answers

dir() returns only the names present in the local scope. Use the locals() function to get the local scope as a dictionary:

for obj in locals().values():
        print(asizeof.asizeof(obj) / 1024)

Note that outside of functions, locals() is the same mapping as globals().

If asizeof() is in the dictionary, you want to filter it out:

for name, obj in locals().items():
    if name != 'asizeof':
        print(asizeof.asizeof(obj) / 1024)

dir() without arguments is functionally equivalent to sorted(locals()) (a sorted list of the keys of the local namespace).

user_1002970

If you prefer to use a standard library and also want them sorted by size:

import sys
objects=[]
for name,obj in locals().items():
   objects.append([name,sys.getsizeof(obj)])
sorted(objects,key=lambda x: x[1],reverse=True)
user_72906020

You can use gc.get_objects() to just fetch all objects tracked by the collector, not just those in a specific namespace. I'd start by using it to count the number of instances of each type as that might give you some clues in itself.

from collections import Counter
c = Counter(type(o) for o in gc.get_objects())
print(c.most_common(20))

Then you might drill down to find the size of any likely suspects.

user_1076600
Related