How to divide a spark dataframe into n different chunks and convert them to dataframe and append into one?

user_8376346 0
Viewed 626

I have a spark dataframe of 100000 rows. Is there a way to loop though 1000 rows and convert them to pandas dataframe using toPandas() and append them into a new dataframe?

Directly changing this by using toPandas() is taking a very long time. There is no column by which we can divide the dataframe in a segmented fraction.

pyspark python
2 Answers

You can directly use limit -

pd_df = ....

sparkDF_1k = sparkDF.limit(1000)

pd_df = pd.concat([pd_df,sparkDF_1k.toPandas()])
user_91089120 
firstThousand = train.limit(1000).collect()[:1000]
df = spark.createDataFrame(firstThousand)
df.toPandas()
user_162361180
Related