I have an RDD in format [(("doc1", ("ab","bc","cd",...)),("doc2", ("de","ab","ab",...)))...]
Can we map this to an RDD of format [((ab, (doc1, doc2,...)),(bc, (doc1, doc2,...))]
I have an RDD in format [(("doc1", ("ab","bc","cd",...)),("doc2", ("de","ab","ab",...)))...]
Can we map this to an RDD of format [((ab, (doc1, doc2,...)),(bc, (doc1, doc2,...))]
I am not sure if your list contains a single tuple or multiple tuples by looking at the code you provided.
Assuming you have an rdd of tuples, you can apply
flatMap to flatten the second element of the original tuple, then applyreduceByKey to aggregate the result into tuples of (key, list), then applymap to convert the inner lists back to tuplesrdd = (sc.parallelize([("doc1", ("ab","bc","cd")), ("doc2", ("de","ab","ab"))])
.flatMap(lambda x: [(i, [x[0]]) for i in x[1]])
.reduceByKey(lambda a, b: a + b)
.map(lambda x: (x[0], tuple(x[1]))))
print(rdd.collect())
[('cd', ('doc1',)), ('bc', ('doc1',)), ('ab', ('doc1', 'doc2', 'doc2')), ('de', ('doc2',))]