For an example,I have two dataframe such like this.
a:
name|age
Ted |15
b:
name|weight
Ted |75
And I use a.withColumn("id",monotonically_increasing_id) to generate a unique column for dataframe a. Then I save dataframe a.
When I use b.join(a,"name"), it is possible to get the wrong id not same as dataframe a.
It doesn't happen steadily. And I found that the error ids are in the same number range.T he first few digits of the number are the same. It looks like a certain range of ids got confused. I have to use zipwithIndex instead.