Memory leak where CPython extension returns a 'PyList_New' instance to Python, which is never deallocated

user_864037 0
Viewed 85

I've been trying to debug a memory leak for a few days and I'm running out of ideas.

High-level: I've written a CPython extension that allows querying against binary data files, and it returns the results as a Python list of objects. Usage is similar to this psuedocode:

for config in configurations:
    s = Strategy(config)
    for date in alldates:
        data = extension.getData(date)
        # do analysis on 'data', capture/save statistics

I've used tracemalloc, memory_profiler, objgraph, sys.getrefcount, and gc.get_referrers to try to find the root cause, and these tools all point to this extension as source of an exorbitant amount of memory (many gigs). For context, a single record in the binary file is 64 bytes, there are typically 390 records per day, so each date iteration is working with ~24K bytes. Now, there are many iterations happening (synchronously), but in each iteration the data is used as a local variable, so I expected each subsequent assignment to deallocate the previous object. The results from memory_profile suggest otherwise...

Line #    Mem usage    Increment  Occurences   Line Contents
============================================================
    86     33.7 MiB     33.7 MiB           1    @profile
    87                                          def evaluate(self, date: int, filterConfidence: bool, limitToMaxPositions: bool, verbose: bool) -> None:
    92    112.7 MiB      0.0 MiB         101        for symbol in self.symbols:
    93    111.7 MiB      0.0 MiB         100            fromdate: int = TradingDays.getAdjacentDay(date, -(self.config.analysisPeriod - 1))
    94    111.7 MiB      0.0 MiB         100            throughdate: int = date
    95                                                  
    96    111.7 MiB      0.0 MiB         100            maxtime: int = self.config.maxTimeToGain
    97    111.7 MiB      0.0 MiB         100            target: float = self.config.profitTarget
    98    111.7 MiB      0.0 MiB         100            islong: bool = self.config.isLongStrategy
    99                                                      
   100    111.7 MiB      0.8 MiB         100            avgtime: Optional[int] = FileStore.getAverageTime(symbol, maxtime, target, islong, fromdate, throughdate, verbose)
   101    111.7 MiB      0.0 MiB         100            if avgtime is None:
   102    110.7 MiB      0.0 MiB          11                continue
   103                                                      
   104    112.7 MiB     78.3 MiB          89            weightedModel: WeightedModel = self.testAverageTimes(symbol, avgtime, fromdate, throughdate)
   105    112.7 MiB      0.0 MiB          89            if weightedModel is not None:
   106    112.7 MiB      0.0 MiB          88                self.watchlist.append(weightedModel)
   107    112.7 MiB      0.0 MiB          88                self.averageTimes[symbol] = avgtime
   108                                                      
   109    112.7 MiB      0.0 MiB           1        if verbose:
   110                                                  print('\nFull Evaluation Results')
   111                                                  print(self.getWatchlistTableString())
   112                                         
   113    112.7 MiB      0.0 MiB           1        self.watchlist.sort(key=WeightedModel.sortKey, reverse=True)
   114                                         
   115    112.7 MiB      0.0 MiB           1        if filterConfidence:
   116    112.7 MiB      0.0 MiB          91            self.watchlist = [ m for m in self.watchlist if m.getConfidence() >= self.config.winRate ]
   117                                                  
   118    112.7 MiB      0.0 MiB           1        if limitToMaxPositions:
   119                                                  self.watchlist = self.watchlist[:self.config.maxPositions]
   120                                         
   121    112.7 MiB      0.0 MiB           1        return

This is from the first iteration of the evaluate function (there are 30 iterations total). Line 104 is where it seems to be accumulating memory. What's strange is that the weightedModel contains only basic stats about the data queried, and that data is stored in a loop-local variable. I can't figure out why the memory used is not cleaned up after each inner iteration.

I've tried to del the objects in question after an iteration completes, but it has no effect. The refcount does seem high for the containing objects, and gc.get_referrers shows an object as referring to itself (?).

I'm happy to provide additional information/code, but I've tried so many things at this point a braindump would be a complete mess :) I'm hoping someone with more experience might be able to help me focus my thought process.

Cheers!

python-3.8 cpython memory-leaks
1 Answers

Found it! The leak was one layer deeper, where the extension function builds an instance of a Python object.

This was the leaky version:

PyObject* obj = PyObject_CallObject(PRICEBAR_CLASS_DEF, args);

PyObject_SetAttrString(obj, "id", PyLong_FromLong(bar->id));
# a bunch of other attrs...

return obj;

This is the fixed version:

PyObject* obj = PyObject_CallObject(PRICEBAR_CLASS_DEF, args);

PyObject* id = PyLong_FromLong(bar->id);
# others...

PyObject_SetAttrString(obj, "id", id);
# others...

Py_DECREF(id);
# others...

return obj;

For some reason I had it in my head that the PyLong_FromLong function did NOT increment the ref count of the resulting object, but this is apparently not true. This is how I wound up with an extra reference count for every bar object that was created.

user_8640370
Related