I need to format the result of my join between two dataframes where the other dataframe has more than one valid value for that index key

Viewed 27

I have two dataframes that I am joining on the index key ['UWI']. I use the following code to join them successfully and then write it to a CSV file to look at the result.

joinedDF = df.join(df1.set_index('New_UWI'), on='New_UWI')
joinedDF.to_csv(r'Joined_Water_Analysis_WithLithium.csv', index=False)

Code works great and one dataframe is created that shows the UWI and its info from the df1 dataframe has been joined correctly to the same UWI in the calling dataframe df. BUT when reviewing the CSV file I realized that if there is more than one record in df1 for a particular [UWI] the Join is forced to replicate that same [UWI] df record in joinedDF. So for example, if df had two rows of valid data for UWI Well 1:

df UWI Formation Top sample depth Bottom sample Depth
Well 1 Upper Devonian 2300 2307
Well 1 Lower Cretaceous 750 757

And df1 had two rows of data for that same UWI Well 1:

df1 UWI Formation Top sample depth Bottom sample depth Lithium
Well 1 Cretaceous 745 759 8.5
Well 1 Upper Devonian 2305 2308 71

The JoinedDF will have four rows. The join needs to attach to each Well 1 in df, the two records from df1 Well 1. The resultant JoinedDF is hard for one to scan and determine which Lithium value should go to which Well 1 row in df. It gets worse when there are ten unique records for Well 1 in df and in df1 there are 2 unique records. The result is 20 rows to match which Lithium sample should go with the appropriate record for Well 1.

df UWI Formation Top sample depth Bottom sample Depth df1 UWI Formation Top sample depth Bottom sample depth Lithium
Well 1 Upper Devonian 2300 2307 Well 1 Cretaceous 745 759 8.5
Well 1 Lower Cretaceous 750 757 Well 1 Upper Devonian 2305 2308 71
Well 1 Upper Devonian 2300 2307 Well 1 Cretaceous 745 759 8.5
Well 1 Lower Cretaceous 750 757 Well 1 Upper Devonian 2305 2308 71

I would prefer if the Join would only show the two original rows for df Well 1 and not replicate the df cell values in the additional rows. In the CSV file I am manually deleting the replicate values in the cells for Well 1 df so the result looks more like below. But is there a way to make Join do this formatting for me because this is a long and tiring manual process otherwise?

df UWI Formation Top sample depth Bottom sample Depth df1 UWI Formation Top sample depth Bottom sample depth Lithium
Well 1 Upper Devonian 2300 2307 Well 1 Cretaceous 745 759 8.5
Well 1 Lower Cretaceous 750 757 Well 1 Upper Devonian 2305 2308 71
Well 1 Cretaceous 745 759 8.5
Well 1 Upper Devonian 2305 2308 71

To make this example short I am only showing that in both Dataframes there are several common columns such as Formation, Top Interval, Bottom Interval, KB elev etc that help us determine which row in df Well 1 should be matched with the unique Lithium value from df1 Well 1.

1 Answers
joinedDF = df.join(df1.set_index('New_UWI'), on='New_UWI')
joinedDF = joinedDF.drop_duplicates()

Or simply

joinedDF = df.join(df1.set_index('New_UWI'), on='New_UWI').drop_duplicates()

Please let me know if this does not work.

Related