How to solve OOM error in Azure Databricks due to using Pandas datasets?

user_18834402 0
Viewed 121

I am using Azure Databricks to process data. I loaded data from Azure blobs and converted it into pandas dataset for further processing, but then I encountered an error:

ConnectException: Connection refused (Connection refused)

ConnectException error: This is often caused by an OOM error that causes the connection to the Python REPL to be closed. Check your query's memory usage

I did some search and found that might caused by using pandas dataset. However, I have no idea how to solve this OOM error. Could anyone please suggest a way with possible steps to guide through? Thank you.

pyspark pandas azure-databricks databricks python
1 Answers

When you collect the data as Pandas dataframe, then the whole dataset is pulled from the worker nodes into the driver node, and this dataset then should fit into the memory, otherwise you'll get OOM issue (see this knowledge base article). To solve this issue, there are different ways:

  • Rethink how you do the data processing - maybe it's possible to implement it using the Spark functions, so it will run in the distributed manner
  • Instead of using Pandas API, look if you can use Pandas API on Spark - then it will be also distributed
  • Select bigger node size for the driver node in the cluster configuration. But you'll get a higher costs (take into account that when you use "normal" Pandas, your worker nodes are idle, but you're paying for them)
Alex_Ott_186270
Related