Merge list of lists in pySpark RDD

user_2929998 0
Viewed 5091

I have lists of tuples that I want to combine into one list. I've been able to process the data using lambdas and list comprehension to where I'm close to being able to use reduceByKey but not sure how to merge the lists. So the format...

[[(0, 14), (0, 24)], [(1, 19), (1, 50)], ...]

And I would like it to be this way....

[(0, 14), (0, 24), (1, 19), (1, 50), ...]

Code that got me to where I need to be...

test = test.map(lambda x: (x[1], [e * local[x[1]] for e in x[0]]))
test = test.map(lambda x: [(x[0], y) for y in x[1]])

But not sure from there what to do to merge the lists

pyspark apache-spark python
2 Answers

You can do,

test = test.flatMap(identity)

or 

test = test.flatMap(lambda list: list)
user_15921910

Thanks to @mrsrinivas for providing the hint...

test = test.flatMap(lambda xs: [(x[0], x[1]) for x in xs])

user_29299980
Related