How to ensure same partitions for same keys while reading two dataframes in pyspark?

Viewed 185

Say I have two dataframes :

data1=[(0, 4, 2),
 (0, 3, 3),
 (0, 5, 2),
 (1, 3, 5),
 (1, 4, 5),
 (1, 5, 1),
 (2, 4, 2),
 (2, 3, 2),
 (2, 1, 5),
 (3, 5, 2),
 (3, 1, 5),
 (3, 4, 2)]
df1=spark.createDataFrame(data1,schema = 'a int,b int,c int')
data2 = [(0, 2, 1), (0, 2, 5), (1, 4, 5), (1, 5, 3), (2, 2, 2), (2, 1, 2)]
df2=spark.createDataFrame(data2,schema = 'a int,b int,c int')

so the dataframes on disk in a csv are like so:

df1.show()
|  a|  b|  c|
+---+---+---+
|  0|  4|  2|
|  0|  3|  3|
|  0|  5|  2|
|  1|  3|  5|
|  1|  4|  5|
|  1|  5|  1|
|  2|  4|  2|
|  2|  3|  2|
|  2|  1|  5|
|  3|  5|  2|
|  3|  1|  5|
|  3|  4|  2|
+---+---+---+

df2.show()
+---+---+---+
|  a|  b|  c|
+---+---+---+
|  0|  2|  1|
|  0|  2|  5|
|  1|  4|  5|
|  1|  5|  3|
|  2|  2|  2|
|  2|  1|  2|
+---+---+---+

I want to read them in and then merge them basis a. I can side-step the shuffle step if I can get same keys (a) on the same partitions. How do I:

  1. Get the same keys on the same node? (I can do df.repartition() after spark.read.csv but, does that not first read the data into whatever partitioning spark sees fit and then repartitions it as I want? This shuffle is what I want to avoid in the first place
  2. Once I succeed in step 1, tell spark that now the keys are on the same node and it does not have to go through the shuffle?
0 Answers
Related