I am trying to perform data loading from A huge csv file, around 8-10gb into SQL Server db using dask, but i am always facing a connection timeout error midway, after loading some 2k records. Below is my error.
A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.\r\n (10060)
Below is my code, I have also mentioned the timeout as 999 seconds, not sure, if it is of any help. please help me overcome the problem. Thanks in advance.
import dask.dataframe as dd
import pyodbc
df = dd.read_csv(r'../csv', dtype = {'col1':'str','col2':'str',..},engine='python', error_bad_lines=False)
cnxn = pyodbc.connect(
Trusted_Connection='Yes',
Driver='{ODBC Driver 17 for SQL Server}',
Server='xx.x.x.xxx',
Database='abcdatabase',
timeout=999.0
)
df = df.fillna(value='')
for index,row in df.iterrows():
cursor.execute('INSERT INTO table col1, col2,..) values(?,?,?..)',
row['col1'],row['col2]...)
cnxn.commit()
cursor.close()
cnxn.close()