pyspark - merge two data frames with extra column in target

Viewed 172

I am trying to do merge like feature in SQL using pyspark. i have below 2 data frames as source and target.

source
    
id,name,city
1,abc,pune
2,xyz,noid

I want below target dataframe with 2 extra column

id,name,city,createdAt,lastupdatedAt
1,abc,pune,current_time,current_time
2,xyz,noida,current_time,current_time

Here key column is id field. So if there is any change in the source data frame my target data frame should reflect the same. if source is having updated data, target field createdAt remains the same and we updated the lastupdatedAt field. If source is having new data, target field createdAt should be updated while the lastupdatedAt field remains the same.

Can someone please help me with the logic or if need to make any design changes to achieve this requirement.

Thanks in advance, Wasim

1 Answers
Related