I am creating a connection, and them am trying to read in the data in chunks and write the values to an identical list. But the end result is double the records in the destination sharepoint list. I am using pandas read_sql() and to_sql() methods with the CDATA connection.
If I do the read and write in one go, I get the same number of records....if I use the chunk loop, I get double (not always double - but always many more, like a 30,000 record source list generating 50,000 target records).
https://cdn.cdata.com/help/RSF/py/default.htm
def add_df_to_spol_list(df, listname, con, index=False):
df.to_sql(listname, con=con, if_exists="append", index=index)
# read in source dataframe using chunks (unreliable)
sql = "SELECT * from [" + prod_list_name + "]"
for chunk in pd.read_sql(sql, prod_engine_spol, chunksize=100):
add_df_to_spol_list(df=chunk, listname=dev_list_name, con=dev_engine_spol)
Does CDATA not respect the "chunking" protocol (if I can call it that).