How to compare values in a specific column between 2 dataframes to check if there's a new value in the newer dataframe column?

user_15938478 0
Viewed 27

I am constantly pulling new data from a website and my goal is to compare the data in a specific column from the last time I pulled data to the data that was newly pulled. If there is a new row (identified by comparing the data in each column, for example, a name column in each dataframe) then take the row associated with the new name and move it to the top of the data in the newly scraped dataframe (reason being that the data on the website I am scraping from isn't in order from newest to oldest). I figured out how to scrape the data, remove columns, and remove rows with certain values from a dataframe, but this has been my struggle. I appreciate all the help you guys have to offer!

Example dataframes:

olddf = pd.DataFrame({'Name': ['apple', 'oranage', 'banana'], 'Value': ['red', 'orange', 'yellow'])
newdf = pd.DataFrame({'Name': ['apple', 'oranage', 'greenapple', 'banana'], 'Value': ['red', 'orange', 'green', 'yellow'])
goaldf = pd.DataFrame({'Name': ['greenapple', 'apple', 'oranage', 'banana'], 'Value': ['green', 'red', 'orange', 'yellow'])

Old example dataframe output:

   Name       Value
0  apple      red
1  orange     orange
2  banana     yellow

New example dataframe output:

   Name       Value
0  apple      red
1  orange     orange
2  greenapple green
3  banana     yellow

Goal example dataframe output:

   Name       Value
0  greenapple green
1  apple      red
2  orange     orange
3  banana     yellow
pandas dataframe python
1 Answers

Use -

common = newdf.merge(olddf,on=['Name','Value'])
pd.concat([newdf[(~newdf.Name.isin(common.Name))&(~newdf.Value.isin(common.Value))], olddf]).reset_index(drop=True)

Output

         Name   Value
0  greenapple   green
1       apple     red
2     oranage  orange
3      banana  yellow
user_40980130
Related