Cannot reach Spark Web UI located inside a Docker container

user_6927126 0
Viewed 4551

I have a remote Virtual Machine and I'm developing a Spark Application that runs inside a Docker container.

2018-12-16 13:07:10 INFO  main [AbstractConnector] - Started ServerConnector@79c3f01f{HTTP/1.1,[http/1.1]}{0.0.0.0:4040}
2018-12-16 13:07:10 INFO  main [Utils] - Successfully started service 'SparkUI' on port 4040.
...
2018-12-16 13:07:10 INFO  main [SparkUI] - Bound SparkUI to 0.0.0.0, and started at http://f58300e7e6ea:4040

The log says that it launched SparkUI correctly, but binded to the container's localhost. At this point i decided to EXPOSE port 4040 during the building phase and to publish it during run with -p 4040:4040/tcp binding the two ports.

When i try to reach <remote host name>:4040 from my local machine (in Firefox) i can't connect to SparkUI. I also tried to telnet it but nothing.

When i start the container i can see the 4040 port listeining using netstat but maybe it is not reachable from remote. How can i manage to do this?

Basically I want to reach SparkUI from my Home PC --> Remote VM --> (Docker Container with Spark UI) using my browser.

The Remote VM runs RHEL 7.

rhel7 centos7 apache-spark docker
2 Answers

In your logs it says that spark UI is started at started at http://f58300e7e6ea:4040 in here f58300e7e6ea is a docker internal network hostname.

So what you have to do is following,

First in your application before deployment set following two configs

  1. spark.driver.bindAddress as the hostname of any string of your choise
  2. spark.driver.host as your Remote VM ip address.

Secondly when you are deploying the docker container using image use --hostname flag to introduce a hostname to the container and use the previously selected string. As example docker run --hostname myHostName --ip 10.1.2.3 ubuntu:16.04

user_71612030

The below command worked for me for pyspark

docker run -p 4040:4040 --hostname localhost -it apache/spark-py /opt/spark/bin/pyspark
user_37820930
Related