Problem when I try to import big database into SQL Azure with Python

user_10765638 0
Viewed 44

I have a pretty weird problem, I am trying to extract with Python a SQL database in Azure. Within this database, there are several tables (I explain this because you gonna see a "for" loop in the code).

I can import some tables without problem, others (the ones that take the longest, I suppose it is because size) fail.

Not only does it throw an error ( [1] 25847 killed / usr / bin / python3 ), but it directly kicks me out of the console.

Does anyone know why? Is there an easier way to calculate the size of the database without import the entire database with pd.read_sql ()?

code:

cnxn = pyodbc.connect('DRIVER={ODBC Driver 17 for SQL Server};SERVER='+server+';DATABASE='+database+';UID='+username+';PWD='+ password)
cursor = cnxn.cursor()
query = "SELECT * FROM INFORMATION_SCHEMA.TABLES"
df = pd.read_sql(query, cnxn)
df


DataConContenido = pd.DataFrame({'Nombre':[], 'TieneCon?':[],'Size':[]})

for tablas in df['TABLE_NAME']:
    cnxn = pyodbc.connect('DRIVER={ODBC Driver 17 for SQL Server};SERVER='+server+';DATABASE='+database+';UID='+username+';PWD='+ password)
    cursor = cnxn.cursor()
    query = "SELECT * FROM " + tablas
    print("vamos con  "+ str(tablas))
    try:
        df = pd.read_sql(query, cnxn)
        size=df.shape
        if size[0] > 0:
            DataConContenido= DataConContenido.append(dict(zip(['Nombre','TieneCon?','Size'],[tablas,True,size])),ignore_index=True)
        else:
            DataConContenido= DataConContenido.append(dict(zip(['Nombre','TieneCon?','Size'],[tablas,False,size])),ignore_index=True)
    except:
        pass

Could it be that the connection drops when it takes so long and that is why the error named above?

pandas azure-sql-database sql python database
1 Answers

I think the process is getting killed in the below line :

DataConContenido= DataConContenido.append(dict(zip(['Nombre','TieneCon?','Size'],[tablas,True,size])),ignore_index=True)

You could double confirm by adding a print statement just above it.

print("Querying Completed...")

You are getting KILLED mainly because there is a probability that your process crossed some limit in the amount of system resources that you are allowed to use. This specific operation to me appears like one.

If possible you could query and append in batches rather than doing in one shot.

user_137552460
Related