How to make python ignore an object for garbage collection?

user_11444902 0
Viewed 1091

At the start of my code, I load in a huge (33GB) pickled object. This object is essentially a huge graph with many interconnected nodes.

Periodically, I run gc.collect(). When I have the huge object loaded in, this takes 100 seconds. When I change my code to not load in the huge object, gc.collect() takes .5 seconds. I assume that this is caused by python checking through every subobject of this object for reference cycles every time I call gc.collect().

I know that neither the huge object, nor any of the objects it references when it is loaded in at the start, will ever need to be garbage collected. How do I tell python this, so that I can avoid the 100s gc time?

garbage-collection python
1 Answers

In python 3.7 you might be able to hack something using https://docs.python.org/3/library/gc.html#gc.freeze

allocate_a_lot()
gc.freeze() # move all objects to a permanent generation. none will be collected
allocate_some_more()
gc.collect() # collect all non-frozen objects
gc.unfreeze() # return to sanity

This said, I think that python does not offer the tools for what you want. In general all garbage collected languages do not want you to do manual memory management.

user_18291550
Related