Do I need to manually repartition a big csv file when I read it with spark.read

user_5877122 0
Viewed 222

I have a big CSV file on my local disk 10GB (no HDFS).

I read it with: 

val df = spark.read.format("csv").load("file:///data/big.csv")

Do I need to repartition (or manually split) it before doing some transformations or actions on it or does spark dataframe will do this for me automatically?

When I was using RDD I needed to repartition it to get more than 1 core working on it but with DF I do not know.

Thanks

apache-spark-sql apache-spark
1 Answers

Spark by default created 200 shuffle partition. So you don't need to do it manually. However to improve performance you can choose to repartition data. However make sure that it will do the full shuffle of table data.

user_25047620
Related