Produce multiple lists from a list of lists in pyspark

Viewed 51

I have some records with the schema below:

    (key,[[value1,value2],[value3,value4]])

What I would like to do is for each of these records emit:

    (key,[value1,value2])
    (key,[value3,value4])

using the RDD API in Pyspark.

For one-to-many relationship between input and output I think I could use flatMap but I am not sure how to apply it in this case. Any ideas?

1 Answers

You can use flatMapValues:

rdd2 = rdd.flatMapValues(lambda x: x)
Related