How do you set a value in RDD once you transform it? I am modifying a JSON file with pyspark and I have this case:
categories: [ {alias, title}, {alias, title}, {alias, title} ]
I have made a transformation that creates a list of titles for each row:
[title, title, title].
But how do I set the result back to the key categories?
At the end I want to get:
categories: [title, title, title]
This is the transformation that I am doing:
restaurantRDD.map(lambda x: x.data).flatMap(lambda x: x).map(lambda x: [row.title for row in x.categories])
There are also multiple transformations from restaurantRDD similar to this one which are modifying some parts of the JSON. How can I apply them both at the same time and then write them in a new JSON file?
Should I use something else instead of RDD?