I am running pyspark program in pycharm local on windows 10 machine . I want to open spark web ui to monitor job and understand metrics showed over spark web ui . While running same code on jupyter I can access web ui but when I run program using pycharm I do not see option to access spark web ui , even I tried url local:4040 or localhost:4041 but , this did not work .
2 Answers
You can configure which port that ui can access in sparksession.
Example:
spark = SparkSession.builder.master("local").\
appName("Word Count").\
config("spark.driver.bindAddress","localhost").\
config("spark.ui.port","4050").\
getOrCreate()
Now once the spark session initialized then we can access spark ui in http://localhost:4050/jobs/ until the pycharm job finished executing.
In case if the port is not able to access at the time then in your log spark prints where does we can access the spark ui.
Sample log:
2020-08-17 14:39:37 WARN Utils:66 - Service 'SparkUI' could not bind on port 4050. Attempting port 4051.
You have to maintain your spark session open.
To keep your session open within python script in Pycharm add input("Press enter to terminate") before your spark.stop().
Until you press enter the session remains open & you can access spark Web UI.
Related