trying to set spark ui port to a range, but it keeps running on a random number

user_18056379 0
Viewed 169

i try to set up the spark ui port to 35000 and max retry 200. but when it runs, it doesn't use the 35xxx ports. it uses a random number ports. i have tried both ways by changing the config and using code (according to How do I change the spark.ui.port?)

does anyone have idea on this?

here is the config spark-defaults.conf.j2:

spark.master yarn
spark.port.maxRetries 200
spark.ui.port 35000
spark.blockManager.port 36000
spark.driver.blockManager.port 37000
spark.driver.port 38000

here are the logs:

-Dspark.ui.port=35000 
... 
INFO - Bound SparkUI to 0.0.0.0, and started at http://xxx.xxx:34138
apache-spark
1 Answers

Try to add

spark.ui.port=4040

as context settings : under your spark-job-server installation/bin file :

settings.conf as :

context-settings {
num-cpu-cores = 4           # Number of cores to allocate.  Required.
memory-per-node = 2g         # Executor memory per node, -Xmx style eg 
spark.ui.port=4040
}
user_103773050
Related