I am trying to do merge like feature in SQL using pyspark. i have below 2 data frames as source and target.
source
id,name,city
1,abc,pune
2,xyz,noid
I want below target dataframe with 2 extra column
id,name,city,createdAt,lastupdatedAt
1,abc,pune,current_time,current_time
2,xyz,noida,current_time,current_time
Here key column is id field. So if there is any change in the source data frame my target data frame should reflect the same. if source is having updated data, target field createdAt remains the same and we updated the lastupdatedAt field. If source is having new data, target field createdAt should be updated while the lastupdatedAt field remains the same.
Can someone please help me with the logic or if need to make any design changes to achieve this requirement.
Thanks in advance, Wasim