Steadily Increasing Memory Usage After Python Scripts Run Through Task Scheduler

Viewed 926

I have several python script scheduled tasks that run on my machine (not an administrator on said machine - so I can't dig in deeper with perfmon or other system tools). Each day I see a fairly constant increase (7% of 16GB of RAM) in system memory usage and I suspect a possible memory leak in my code.

A complete restart of my machine results in the memory clearing. After about 9-10 days my system memory is maxed out again requring another restart.

Each script does approximately the same function:

  • Reach out to a network SQL db using sqlalchemy package
  • Create a pandas dataframe from retrieved DB return
  • merge pandas dataframe with existing excel .XLS file and save over existing XLS file
  • import the saved XLS file and style it (color, column width,etc.) with the Openpyxl package

After I am through with each respective object, I am deleting it like so:

del (sample_df, previous_df)

At the end of my script I am also including the following code to catch any global vars I have missed and manually running the garbage collector:

for name in dir():
    if not name.startswith('_'):
        del globals()[name]

import gc
gc.collect()

I have attempted to utilize the Pympler package and its .tracker capabilities and at the end of my script running

tracker.print_diff()

results in the following objects (left?) in memory that appear to have been uncollected by the garbage collector:

                  types |   # objects |   total size

                  list |       19018 |    968.68 KB
                   str |       19025 |    862.44 KB
                   int |        1122 |     13.15 KB

After garbage collector is run I have also looked at the memory usage of items still in the local namespace:

import sys
for var, obj in locals().items():
   print var, sys.getsizeof(obj)

Returns:

__ 36
__builtin__ 28
SummaryTracker 452
gc 28
__package__ 8
_i11 78
__doc__ 85
_i 66
__file__ 208
_sh 28
name 23
_7 256
_6 36
_3 3212
_2 36
_i13 190
_i12 66
_iii 66
_i10 66
_oh 140
_dh 40
_i9 110
_i8 390
_i7 36
_i6 32
_i5 390
_i4 190
_i3 42
_i2 38
_i1 390
tracker 32
__builtins__ 28
sys 28
__name__ 29
___ 3212
_ 256
_ii 78
_ih 100

The "___" object appears to be using decent memory, but I have no idea what that is from.

I've tried to include as much troubleshooting as I have done. Thanks in advance for your help.

0 Answers
Related