How to set partition count while reading CSV extract using SparkSession?

user_10187977 0
Viewed 49

I'm reading a CSV extract using sparkSession and the framework is trying to create 10K partitions/tasks because of this huge task count, the spark job was failing with java.lang.OutOfMemoryError: GC overhead limit exceeded.

Below is the sample code which is creating 10K tasks while reading the extract:

sparkSession
    .read
    .csv(dfwAbsHdfsPath)

Is there any way I can reduce or configure the tasks/partitions count? Just to add, we are using Spark version 2.3.1.

apache-spark-dataset apache-spark-sql apache-spark
1 Answers

You can change the default parallelism in the config like this:

.config("spark.default.parallelism", NUMBER)

This influences the partitioning.

user_126381180
Related