Pyspark: An error occurred while trying to connect to the Java server

Viewed 97

I am starting my Pyspark session in JupyterNotebook:

spark = (SparkSession.builder.appName('myApp')
         .config("spark.executor.memory", "4g")
         .config("spark.executor.instances", "20")
         .config("spark.master", "yarn")
         .config("spark.driver.memory", "8g")
         .config("spark.ui.port", "4373")
         .config("spark.driver.maxResultSize", "1g")
         .enableHiveSupport()
        .getOrCreate())

But then got this error:

Py4JNetworkError: An error occurred while trying 
to connect to the Java server (127.0.0.1:34032)

Sometimes the SparkSession would start fine but then later on when I am working on the script it shows the error. Would like to know how to fix it.

0 Answers
Related