I am starting my Pyspark session in JupyterNotebook:
spark = (SparkSession.builder.appName('myApp')
.config("spark.executor.memory", "4g")
.config("spark.executor.instances", "20")
.config("spark.master", "yarn")
.config("spark.driver.memory", "8g")
.config("spark.ui.port", "4373")
.config("spark.driver.maxResultSize", "1g")
.enableHiveSupport()
.getOrCreate())
But then got this error:
Py4JNetworkError: An error occurred while trying
to connect to the Java server (127.0.0.1:34032)
Sometimes the SparkSession would start fine but then later on when I am working on the script it shows the error.
Would like to know how to fix it.