I have 20 slow running SQL queries against 20 different tables a SQL Server database.
The code looks something like:
df01 = pd.read_sql(q01, con=cnxn)
df02 = pd.read_sql(q02, con=cnxn)
...
df20 = pd.read_sql(q20, con=cnxn)
Where there is very little overlap between the data being fetched.
I'm looking for a good way to run these queries simultaneously... right now my application hangs for ~10min while it fetches df1 to df20.
I'm not sure whether async/multithreading/multiprocessing is most appropriate. Some info:
- The Python application is essentially idle while the data is being fetched (CPU ussage ~1%)
- The the network is also not saturated (the data returned into each
dfis only a few thousand rows) - The database is also not saturated because any one query can't leverage all the vCores available.
Any ideas on how to approach this?
PS. I'm using Python 3.9.7