PySpark driver memory exceptions while reading too many small files

userviJp 0
Viewed 24

PySpark version: 2.3.0, HDP 2.6.5

My source is populating a Hive table(HDFS backed) with 826 paritions and 1557242 small files(40KB). I know this is highly inefficient way to store data but I dont have control over my source

The problem now is when I need to do some historical load and I need to scan all the files at once the driver is having memory exceptions. Tried setting driver-memory to 8g,16g and similar configuration for driver.memory.overhead too. But the problem still persists.

What makes me wonder is this is failing in listing files I presume this is just metadata. Is there an explanation why file metadata would need so much memory?

py4j.protocol.Py4JJavaError: An error occurred while calling o351.saveAsTable.
: java.lang.OutOfMemoryError: GC overhead limit exceeded
    at java.lang.String.substring(String.java:1956)
    at java.net.URI$Parser.substring(URI.java:2869)
    at java.net.URI$Parser.parse(URI.java:3049)
    at java.net.URI.<init>(URI.java:746)
    at org.apache.hadoop.fs.Path.initialize(Path.java:202)
    at org.apache.hadoop.fs.Path.<init>(Path.java:171)
    at org.apache.spark.sql.execution.datasources.InMemoryFileIndex$$anonfun$org$apache$spark$sql$execution$datasources$InMemoryFileIndex$$bulkListLeafFiles$3$$anonfun$7.apply(InMemoryFileIndex.scala:251)
apache-spark
1 Answers

It would be helpfull if you can share the parameters you were passing in spark submit.

i too facing similar issue.. adjusting the parameters it worked.

try diffrent configs ( as number s i cant suggest it depends on server configs)

mine :

spark-submit \
  --master yarn \
  --deploy-mode client \
  --driver-memory 5g \
  --executor-memory 6g \
  --executor-cores 3
kartheek0
Related