How to write pyspark dataframe to HDFS and then how to read it back into dataframe?

Viewed 28306

I have a very big pyspark dataframe. So I want to perform pre processing on subsets of it and then store them to hdfs. Later I want to read all of them and merge together. Thanks.

1 Answers
Related