Using a map for tuple in pyspark

Viewed 343

I have an RDD in format [(("doc1", ("ab","bc","cd",...)),("doc2", ("de","ab","ab",...)))...]

Can we map this to an RDD of format [((ab, (doc1, doc2,...)),(bc, (doc1, doc2,...))]

1 Answers

I am not sure if your list contains a single tuple or multiple tuples by looking at the code you provided.
Assuming you have an rdd of tuples, you can apply

  1. flatMap to flatten the second element of the original tuple, then apply
  2. reduceByKey to aggregate the result into tuples of (key, list), then apply
  3. map to convert the inner lists back to tuples
rdd = (sc.parallelize([("doc1", ("ab","bc","cd")), ("doc2", ("de","ab","ab"))])
       .flatMap(lambda x: [(i, [x[0]]) for i in x[1]])
       .reduceByKey(lambda a, b: a + b)
       .map(lambda x: (x[0], tuple(x[1]))))
print(rdd.collect())
[('cd', ('doc1',)), ('bc', ('doc1',)), ('ab', ('doc1', 'doc2', 'doc2')), ('de', ('doc2',))]
Related