I want to know among union() and full_outer_join in spark which is best in term of performance wise. To demonstrate consider the below 2 table:
On full Outer Join:
and on Union() also same result: 
Note: I have tested the performance on more than 1,00,000 rows. what I observed is full outer join on such big dataframe is performing better than union() but I want to know why its happing so? as we already know that union() doesn't do any shuffling of data but join operation in spark does shuffling. Hence, join should be more expensive.
Also I noticed that on less number of dataset, union() and full outer join are performing almost same.
Any help/ insight would be greatful.