Spark Python/SQL - how to group with unique combinations

user_17603765 0
Viewed 77

I have a 2 column Spark dataframe with unique combinations looking like this (simple version):

col1    col2
   a       b
   b       c
   b       d
   d       e
   f       g

I would like to form groups, in this case there would be 2 groups: (a,b,c,d,e) and (f,g) as these are linked through the combinations. The easiest way to count the number of members of the group would be if I could get this as an output:

col1    col2
   a       b
   a       c
   a       d
   a       e
   f       g

Does anyone know how to get to this? Thank you in advance for your help!

pyspark apache-spark sql python
1 Answers

You need to look into graphs with pyspark and use connectedComponents() which will show groupings.

user_69339930
Related