I have 2 Spark dataframes with same number of columns.
DF1:
ID KEY
1 A
1 A
2 B
3 C
3 C
DF2:
ID KEY
1 A
1 A
1 A
2 B
3 C
3 C
4 D
5 E
5 E
I want to compare these 2 dataframes and write those records that are there in DF2 but not in DF1.
Expected output:
ID KEY
1 A
4 D
5 E
5 E
use .exceptAll function.
`Example:
df1.show()
#+---+---+
#| ID|KEY|
#+---+---+
#| 1| A|
#| 1| A|
#| 2| B|
#| 3| c|
#| 3| c|
#+---+---+
df2.show()
#+---+---+
#| ID|KEY|
#+---+---+
#| 1| A|
#| 1| A|
#| 1| A|
#| 2| B|
#| 3| c|
#| 3| c|
#| 4| D|
#| 5| E|
#| 5| E|
#+---+---+
df2.exceptAll(df1).orderBy("ID").show()
#+---+---+
#| ID|KEY|
#+---+---+
#| 1| A|
#| 4| D|
#| 5| E|
#| 5| E|
#+---+---+