I am trying to change the host of my Spark UI dashboard to 0.0.0.0 on Python.
Currently I have the following to change the port - however, I haven't been able to find the argument which changes the host
from pyspark import SparkContext, SparkConf
conf = SparkConf().set('spark.ui.port', '8787')
sc = SparkContext(conf=conf)
You should create a SparkSession which will create you a SparkContext. I invite you to follow this article explaining the difference between both : SparkSession vs SparkContext
As you said, we can change the port with the key : spark.ui.port
We can do the same with the host address : spark.driver.bindAddress
You can find here more information about which parameters you can add in your configuration.
from pyspark.sql import SparkSession
spark = SparkSession.builder.appName('MyApp').config("spark.driver.bindAddress","localhost").config("spark.ui.port","5372").getOrCreate()
input("Press enter to terminate")
With this lines of code you can access the Spark UI at : http://localhost:5372
Because I added as config parameter :
| key | value |
|---|---|
| spark.driver.bindAddress | localhost |
Here, you import SparkSession from the pyspark.sql package then you create a SparkSession and you wait for the next order with the input I added otherwise the process automatically stop after each run (change this as you wish, I just wrote this input as education purpose only).
Comment if you need more info. Hope I was able to help some.