I have several python script scheduled tasks that run on my machine (not an administrator on said machine - so I can't dig in deeper with perfmon or other system tools). Each day I see a fairly constant increase (7% of 16GB of RAM) in system memory usage and I suspect a possible memory leak in my code.
A complete restart of my machine results in the memory clearing. After about 9-10 days my system memory is maxed out again requring another restart.
Each script does approximately the same function:
- Reach out to a network SQL db using sqlalchemy package
- Create a pandas dataframe from retrieved DB return
- merge pandas dataframe with existing excel .XLS file and save over existing XLS file
- import the saved XLS file and style it (color, column width,etc.) with the Openpyxl package
After I am through with each respective object, I am deleting it like so:
del (sample_df, previous_df)
At the end of my script I am also including the following code to catch any global vars I have missed and manually running the garbage collector:
for name in dir():
if not name.startswith('_'):
del globals()[name]
import gc
gc.collect()
I have attempted to utilize the Pympler package and its .tracker capabilities and at the end of my script running
tracker.print_diff()
results in the following objects (left?) in memory that appear to have been uncollected by the garbage collector:
types | # objects | total size
list | 19018 | 968.68 KB
str | 19025 | 862.44 KB
int | 1122 | 13.15 KB
After garbage collector is run I have also looked at the memory usage of items still in the local namespace:
import sys
for var, obj in locals().items():
print var, sys.getsizeof(obj)
Returns:
__ 36
__builtin__ 28
SummaryTracker 452
gc 28
__package__ 8
_i11 78
__doc__ 85
_i 66
__file__ 208
_sh 28
name 23
_7 256
_6 36
_3 3212
_2 36
_i13 190
_i12 66
_iii 66
_i10 66
_oh 140
_dh 40
_i9 110
_i8 390
_i7 36
_i6 32
_i5 390
_i4 190
_i3 42
_i2 38
_i1 390
tracker 32
__builtins__ 28
sys 28
__name__ 29
___ 3212
_ 256
_ii 78
_ih 100
The "___" object appears to be using decent memory, but I have no idea what that is from.
I've tried to include as much troubleshooting as I have done. Thanks in advance for your help.