How can i get a Cartesian product filtering out the pair of tuples with repeated results?

user_15068341 0
Viewed 55

Im trying to define a function that gets the cartesian product of a given list with itself , however i nedd to filter out the elemnts that contains the same pairs.

For example: Getting the cartesian product of rdd and fiter out the results ((1,0),(1,0)),((2,0),(2,0)) and ((3,0),(3,0))

rdd = sc.parallelize([(1,0), (2,0), (3,0)])


def get_cart(rdd):
   
    a=sorted(rdd.cartesian(rdd).collect()) 
    aRDD=sc.parallelize(a)
 
    return aRDD

Im expecting to get the output:

[((1, 0), (2, 0)), ((1, 0), (3, 0)), ((2, 0), (1, 0)), ((2, 0), (3, 0)), ((3, 0), (1, 0)), ((3, 0), (2, 0))]

Instead im getting:

[((1, 0), (1, 0)),
 ((1, 0), (2, 0)),
 ((1, 0), (3, 0)),
 ((2, 0), (1, 0)),
 ((2, 0), (2, 0)),
 ((2, 0), (3, 0)),
 ((3, 0), (1, 0)),
 ((3, 0), (2, 0)),
 ((3, 0), (3, 0))]
pyspark filtering cartesian-product python
1 Answers

You can use a filter to remove RDD elements with the first tuple equal to the second tuple:

def get_cart(rdd):
    return rdd.cartesian(rdd).filter(lambda r: r[0] != r[1])

Note that there is no need to collect the RDD and re-parallelize it after cartesian, which already returns an RDD.

user_141657300
Related