Databricks Exception: Total size of serialized results is bigger than spark.driver.maxResultsSize

user_9669142 0
Viewed 11464

I'm running a code in Apache Spark on Azure that converts over 3 million XML-files into one CSV-file. I get the following error when I want to do this:

org.apache.spark.SparkException: Job aborted due to stage failure: Total size of serialized results of 1408098 tasks (4.0 GB) is bigger than spark.driver.maxResultSize (4.0 GB)

I know what the error means in general, but I don't know what it means in my case and I don't understand how to solve this.

The code is:

All XML files are loaded:

df = spark.read.format('com.databricks.spark.xml').option("rowTag", "ns0:TicketScan").load('LOCATION/*.xml')

All loaded files are put into a CSV-file:

 def saveDfToCsv(df, tsvOutput):
  tmpParquetDir = "dbfs:/tmp/mart1.tmp.csv"
  dbutils.fs.rm(tmpParquetDir, True)
  df.repartition(1).write.format("com.databricks.spark.csv").option("header", "true").save(tmpParquetDir)
  src = filter(lambda x: "part-00000" in x.name, dbutils.fs.ls('dbfs:/tmp/mart1.tmp.csv'))[0].path
  dbutils.fs.mv(src, tsvOutput)

saveDfToCsv(df, 'LOCATION/database.csv')

I hope my question is clear enough. If not, please allow me to explain it further.

I hope someone can help me.

Best regards.

databricks apache-spark azure python
3 Answers

You need to change this parameter in the cluster configuration. Go into the cluster settings, under Advanced select spark and paste spark.driver.maxResultSize 0 (for unlimited) or whatever the value suits you. Using 0 is not recommended. You should optimize the job by re partitioning.

user_56122150

Looks like your driver have a limited size for storing the result and your resulting files have cross the limit,so you can increase the size of result by the following command in your notebook.

sqlContext.getConf("spark.driver.maxResultSize")
res19: String = 20g

It gives the current max size of storage capacity as 20 GB, mine

sqlContext.setConf("spark.driver.maxResultSize","30g")

To increase the maxResultSize you can use the above command.

It's not recommended because its reduce the performance size of your cluster because then you have minimize the free space allocate to the temporary files for processing in the cluster.But i think it solved your issue.

user_80313500

You need to increase the maxResultSize value for the cluster.

The maxResultSize must be set BEFORE the cluster is started -- trying to set the maxResultSize in the notebook after the cluster is started will not work.

"Edit" the cluster and set the value in the "Spark Config" section under "Advanced Options".

Here is a screenshot of Configure Cluster for Databricks in AWS, but something similar probably exists for Databricks in Azure.

cluster configuration

In your notebook you can verify that the value is already set by including the following command:

enter image description here

Of course 8g may not be large enough in your case, so keep increasing it until the problem goes away -- or something else blows up! Best of luck.

Note: When I ran into this problem my notebook was attempting to write to S3, not directly trying to "collect" the data, so-to-speak.

user_30776730
Related