I am creating a python script to work with Binance API (as my research project). The script uses pandas and bta-lib packages for technical analysis. The following is the main scheme of the program
- The program runs in a loop. Every time, new kline data is received, the data is stored in a sqlite database
- The script then retrieves the last 1000 rows from database and convert it into a pandas dataframe
- Technical analysis is performed on the pandas dataframe using bta-lib package
- The results (last line) from the technical analysis are stored in a separate database.
The problem I am running in is that with each iteration of the loop, the memory consumption of the script keeps on increasing. Using tracemalloc I found that the problem lies with the pandas dataframe (unable to figure out the exact problem though).
C:\Users\Dr Ahsan Iqbal\AppData\Local\Programs\Python\Python38-32\lib\site-packages\pandas\core\internals\blocks.py:643: size=122 MiB, count=6401, average=19.6 KiB C:\Users\Dr Ahsan Iqbal\AppData\Local\Programs\Python\Python38-32\lib\site-packages\pandas\core\internals\managers.py:2132: size=73.4 MiB, count=6400, average=11.7 KiB C:\Users\Dr Ahsan Iqbal\AppData\Local\Programs\Python\Python38-32\lib\site-packages\pandas\core\dtypes\cast.py:1916: size=24.6 MiB, count=6400, average=4024 B main4.py:56: size=7914 KiB, count=207599, average=39 B main4.py:47: size=7899 KiB, count=207597, average=39 B C:\Users\Dr Ahsan Iqbal\AppData\Local\Programs\Python\Python38-32\lib\site-packages\pandas\core\internals\managers.py:1733: size=576 KiB, count=25649, average=23 B C:\Users\Dr Ahsan Iqbal\AppData\Local\Programs\Python\Python38-32\lib\site-packages\pandas\core\generic.py:247: size=500 KiB, count=12811, average=40 B C:\Users\Dr Ahsan Iqbal\AppData\Local\Programs\Python\Python38-32\lib\site-packages\pandas\core\internals\managers.py:1853: size=438 KiB, count=12801, average=35 B C:\Users\Dr Ahsan Iqbal\AppData\Local\Programs\Python\Python38-32\lib\site-packages\pandas\core\flags.py:48: size=375 KiB, count=12811, average=30 B C:\Users\Dr Ahsan Iqbal\AppData\Local\Programs\Python\Python38-32\lib\site-packages\btalib\meta\lines.py:432: size=375 KiB, count=12800, average=30 B
The sizes above (shown in bold) are after getting 1600 lines and they keep on increasing, along with the memory usage of the program (as noticed from tasks manager) and finally the program terminates with runtime error.
The following are the relevant code snippets from the program
def create_ta_db(klines_dict):
for table_name in klines_dict.keys():
db_connect_kline_data = sqlite.connect('kline_data.db')
cursor_kline_data = db_connect_kline_data.cursor()
query = 'SELECT open_time,open_price,high_price,low_price,close_price,volume,close_time,open_time_formatted,close_time_formatted FROM (SELECT * FROM {} ORDER BY id DESC LIMIT 1000) ORDER BY id ASC'.format(table_name)
cursor_kline_data.execute(query)
retrieved_data = cursor_kline_data.fetchall()
df = pd.DataFrame(retrieved_data)
df = df.rename(columns={0: "df_index", 1: "open",2:"high",3:"low",4:"close",5:"volume",6:"close_time",7:"open_time_formatted",8:"close_time_formatted"})
df["open_time"] = df["df_index"].copy()
df = df.set_index("df_index")
sma = btalib.sma(df,period = 1000)
df["sma"] = sma.df["sma"].copy()
df = df.iloc[-1:]
df_list = df.to_dict('records')
df_dict = df_list[0]
del (df)
The function create_ta_db is run in a While loop. What am I doing wrong here ? I have tried gc.collect() but it does not help.