Hi does somebody has any troubleshooting ideas to solve this problem?
I have a standard python-sql connection at my local machine:
from sqlalchemy import create_engine
engine = create_engine("mysql+pymysql://root:*******@localhost/my_DB")
con = engine.connect()
this DB consists of 200+ tables where I store stock/market information and I need to update it daily, in order to do that I usually construct a loop through all the tables to fetch up to date information from yahoo_finance using pandas datareader.
Once loaded into a new DF I use
df_new.to_sql(name = stock_ticker, con = con, if_exists = 'replace', index = False)
to save the new table into my DB.
The code above works just fine when I execute one by one, but when I try to implement the same idea on a loop it just breaks, sometimes on the very first instance of the loop:
for stock in Stocks:
df_new = yahoo_quote(stock)
df_new.to_sql(name = stock_ticker, con = con, if_exists = 'replace', index = False)
My first thought was that somehow I was exhuasting my machine/sql with so many calls, so I tried to add a time.sleep(5) and make sure I erased all the information from memory on each instance, but none of that seems to work. And, as I said, sometimes the computer just breaks on the very first loop.
By "break" I mean that it just keeps running forever without saving the table, usually it takes little less than 1 second to save a table, but when this happens I can leave it running for 10+ minutes and it still won't save it.