I need to connect to an existing SQLite database, compare a key column's value to the value in a dataframe. For every key match between database and dataframe, update the value of a specific column in that row. If a key exists in the dataframe, but not in the database, append the respective row to database. The targets are relatively large datasets, so memory usage and performance is a concern (can be 20-60 gb db, @ ~20 columns and millions of rows).
I've previously tried reading the database into a dataframe and merging the old and new dataframes in memory, but that proving expensive (often a 5 gig dataset grows into 20 gigs in memory).
I'm getting lost in the logic here, and this is the farthest I've gone:
def update_column(tablename, key_value):
c.execute('SELECT key FROM {}'.format(tablename))
for row in c.fetchall():
# populating this key value per row is challenging for me
if row == key_value:
c.execute('UPDATE {} SET last_seen = {} WHERE UUID = {}}'.format(tablename, hunt_date, key_value))
else:
df.to_sql(table_name, if_exists='append')
for index, row in reader.iterrows():
key_value = row['key']
update_column(tablename, key_value)
Example Dataset:
Database
Key First_Seen Last_Seen Data1 Data2
Bigfoot 2015 2015 Blah Blah
Loch_Ness 2016 2016 Blah Blah
UFO 2016 2004 Blah Blah
Dataframe with new data:
Key First_Seen Last_Seen Data Data
UFO 2017 2017 Blah Blah
Tupac 2017 2017 Blah Blah
Desired output in database:
Key First_Seen Last_Seen Data Data
Bigfoot 2015 2015 Blah Blah
Loch_Ness 2016 2016 Blah Blah
UFO 2016 2017 Blah Blah
Tupac 2017 2017 Blah Blah