The spark driver has stopped unexpectedly and is restarting. Your notebook will be automatically reattached

user_11622712 0
Viewed 10024

I try to analyze a dataset of 500Mb in Databricks. These data are stored in Excel file. The first thing that I did was to install Spark Excel package com.crealytics.spark.excel from Maven (last version - 0.11.1).

These are the parameters of the cluster:

Then I executed the following code in Scala notebook:

val df_spc = spark.read
          .format("com.crealytics.spark.excel")
          .option("useHeader", "true")
          .load("dbfs:/FileStore/tables/test.xlsx")

But I got error about the Java heap size and then I get another error "java.io.IOException: GC overhead limit exceeded". Then I executed this code again and got another error after 5 minutes running:

The spark driver has stopped unexpectedly and is restarting. Your notebook will be automatically reattached.

I do not understand why it happens. In fact the data set is quite small for the distributed computing and the cluster size should be ok to process these data. What should I check to solve it?

azure-databricks apache-spark excel scala
2 Answers

I also got stuck in same situation where i am unable to process my 35000 record xlsx file. Below solutions I tried to work around:

  1. With the free azure subscription and 14 day pay as you go mode, you can process xlsx with less number of records.In my case with trial version, I have to change it to 25 records.

  2. Also downgrade the worker type to Standard_F4S 8GB Memory 4core, 0.5DBU, 1 worker configuration.

  3. Added below options:

    sqlContext.read.format("com.crealytics.spark.excel"). option("location","filename here...").option("useHeader","true").option("treatEmptyValueAsNulls","true").option("maxRowsInMemory",20).option("inferSchema","true").load("filename here...")

user_10592300

I had this same issue. We reached out to DataBricks, who provided us this answer "In the past we were able to address this issue by simply restarting a cluster that has been up for a long period of time. This issue occurs due the fact that JVMs reuse the memory locations too many times and start misbehaving."

user_185800090
Related