ConnectException: Connection refused (Connection refused) Error while obtaining a new communication channel

user_18912456 0
Viewed 403

I came across this error and I'm not sure where I've gone wrong! It would be really helpful if someone could help me out on this. Thanks!

Create Output Model Directory

dbutils.fs.mkdirs('/CIFAR10/models/')
Out[21]: True


from keras.models import load_model
modelpath = '/dbfs/CIFAR10/models/cifar_2000pictures.h5'
model2000.save(modelpath)​
#model = load_model(modelpath)


ConnectException: Connection refused (Connection refused)
Error while obtaining a new communication channel
ConnectException error: This is often caused by an OOM error that causes the connection to the Python REPL to be closed. Check your query's memory usage.
Spark tip settings
tf.keras keras azure-databricks databricks deep-learning
1 Answers

OOM (Out Of Memory) issue, Drive crashed out of memory it's not able to establish a new connection with the driver. Please try the below options:

  1. Reduce a load of executors by filtering as much data as possible, Use partition columns. if possible, it will largely decrease the movement of data.
  2. Incorrect Configuration here is a possibility that the application fails due to a YARN memory overhead issue (if Spark is running on YARN).You can resolve it by setting the partition size: Increase the value of spark.sql.shuffle.partitions.

Why is this happening?

No matter how big the cluster is. If you are working on dealing with larger datasets, through python pandas that run operations on a single machine, not like spark runs on multiple machines. So, you can increase the memory size of the driver node.

For more information follow this article:

https://medium.com/disney-streaming/a-step-by-step-guide-for-debugging-memory-leaks-in-spark-applications-e0dd05118958

https://blog.clairvoyantsoft.com/apache-spark-out-of-memory-issue-b63c7987fff

user_182299800
Related