Updating SQLite DB with dataframe using conditional to change column value or append new row

Viewed 4614

I need to connect to an existing SQLite database, compare a key column's value to the value in a dataframe. For every key match between database and dataframe, update the value of a specific column in that row. If a key exists in the dataframe, but not in the database, append the respective row to database. The targets are relatively large datasets, so memory usage and performance is a concern (can be 20-60 gb db, @ ~20 columns and millions of rows).

I've previously tried reading the database into a dataframe and merging the old and new dataframes in memory, but that proving expensive (often a 5 gig dataset grows into 20 gigs in memory).

I'm getting lost in the logic here, and this is the farthest I've gone:

def update_column(tablename, key_value):
    c.execute('SELECT key FROM {}'.format(tablename))
    for row in c.fetchall():
        # populating this key value per row is challenging for me
        if row == key_value: 
            c.execute('UPDATE {} SET last_seen = {} WHERE UUID = {}}'.format(tablename, hunt_date, key_value))
        else:
            df.to_sql(table_name, if_exists='append')

for index, row in reader.iterrows():
    key_value = row['key']
    update_column(tablename, key_value)

Example Dataset:

Database

Key       First_Seen Last_Seen Data1  Data2
Bigfoot   2015       2015      Blah   Blah
Loch_Ness 2016       2016      Blah   Blah
UFO       2016       2004      Blah   Blah     

Dataframe with new data:

Key       First_Seen Last_Seen Data  Data
UFO       2017       2017      Blah  Blah
Tupac     2017       2017      Blah  Blah

Desired output in database:

Key       First_Seen Last_Seen Data  Data
Bigfoot   2015       2015      Blah  Blah
Loch_Ness 2016       2016      Blah  Blah
UFO       2016       2017      Blah  Blah
Tupac     2017       2017      Blah  Blah
2 Answers
Related