Databricks notebook detaches in standard cluster mode

user_9642938 0
Viewed 3076

EDIT: Update. This happens regardless of the number of users. Even with one user, it still happens.

The databricks notebook is repeatedly detaching while in use.

Our data scientist comes from a Jupyter background, and since Koalas has some gaps, he's continued to use Pandas with a few workarounds, which has meant much heavier load on the driver, but the notebook seems functional.

I've looked for people experiencing similar issues, but have not seen this.

This is my spark config:

spark.driver.extraJavaOptions -XX:+UseG1GC
spark.driver.cores 8
spark.driver.memory 16g
spark.executor.extraJavaOptions -XX:+UseG1GC

The two errors the data scientist sees when it detaches:

Notebook detached

Notebook detached
Exception when creating execution context: 
java.util.concurrent.TimeoutException: Exchange timed out after 15 seconds.

Spark context stopped

The spark context has been stopped or the cluster has been terminated.
Please restart the cluster or attach this notebook to a different cluster.

The cluster seems fine, in the sense that I can connect another notebook and run commands.

What I expect to happen is that at no point will the notebook be randomly detached and lose all work.

When I look through the driver logs, these are the only messages that stand out:

19/10/08 18:02:59 INFO TaskSchedulerImpl: Killing all running tasks in stage 82: Stage finished
19/10/08 18:02:59 INFO DAGScheduler: Job 57 finished: collectResult at OutputAggregator.scala:149, took 9.157699 s
19/10/08 18:02:59 INFO SQLAppStatusListener: Execution ID: 28 Total Executor Run Time: 21250
19/10/08 18:02:59 INFO CodeGenerator: Code generated in 21.921114 ms
19/10/08 18:03:00 INFO ProgressReporter$: Removed result fetcher for 8919779546758574174_8732072469296650198_763335e3d46b4641ba75b3c6d4b4ffac
19/10/08 18:04:30 INFO DriverCorral$: Cleaning the wrapper ReplId-5231d-7b5c0-a6423-e (currently in status Idle(ReplId-5231d-7b5c0-a6423-e))
19/10/08 18:04:30 INFO DriverCorral$: sending shutdown signal for REPL ReplId-5231d-7b5c0-a6423-e
19/10/08 18:04:31 INFO PythonDriverLocal$Watchdog: Python shell exit code: 143
19/10/08 18:04:31 INFO PythonDriverLocal$RedirectThread: Python RedirectThread exit
19/10/08 18:04:31 INFO PythonDriverLocal$RedirectThread: Python RedirectThread exit
19/10/08 18:04:31 INFO PythonDriverLocal$Watchdog: No strace information recovered: /tmp/637654b25044473abae9a282b9564078.strace is missing
19/10/08 18:04:31 INFO DriverCorral$: sending the interrupt signal for REPL ReplId-5231d-7b5c0-a6423-e
19/10/08 18:04:31 INFO DriverCorral$: waiting for localThread to stop for REPL ReplId-5231d-7b5c0-a6423-e
19/10/08 18:04:31 INFO DriverCorral$: ReplId-5231d-7b5c0-a6423-e successfully discarded
azure-databricks databricks apache-spark
2 Answers

I had a similar problem. The driver crashes and stops and this message is shown. In my case, the work was being executed just on the driver and it hasn't enough capacity to execute the job (a query + s3 file saving). I decreased the df maxRecordsPerFile so there was more files now (before I had just one) and it was possible to parallelize it between the nodes. After this, the job executed fine. And the driver didn't crash and detached from the notebook anymore. I hope it helps.

user_78859030

I also had a similar problem but reading lines from a single file or from Kafka topics. Our first load had around 2million records. Even after it gets completed, if we received only a new single row, it didn't proceed because delayed streamings were in the "processing queue" (like the "updated 4 hours ago" presented in the picture below).

In these cases, we used "maxBytesPerTrigger" reducing its value until works without breaking or spending a long time to start the streaming queries. We got better results after it.

Without this option or with big values as parameters we got long streaming updates intervals or lost connection errors like this:

enter image description here

user_39156430
Related