How can I access spark web UI using GCP instance?

user_13108817 0
Viewed 485

I'm using spark on Google Cloud Platform instance(hana express).

I installed spark and run spark shell, then shell is well running but I can't access spark web UI.
I added fire wall rules to instance but still doesn't work.
I added screen shot.

Thank you.

apache-spark google-cloud-platform
1 Answers

Have a look at the console messages: 

...
Spark context Web UI available at http://sap-hanaexpress-serverinclapps-1-vm.c.hana.271411.internal:4040
...

You're not able to reach Web UI running at http://sap-hanaexpress-serverinclapps-1-vm.c.hana.271411.internal:4040 from your remote PC. As it was mentioned by @Lamanus this record is for internal usage only. Have a look at the documentation Internal DNS:

Virtual Private Cloud networks on Google Cloud have an internal DNS service that lets instances in the same network access each other by using internal DNS names. Internal A records for virtual machine (VM) instances are created in a DNS zone for .internal. PTR records for VM instances are created in corresponding reverse zones. As you manage your instances, Google Cloud automatically creates, updates, and removes these DNS records.

and

The internal DNS name of a VM instance only resolves to its primary internal IP address. Internal DNS names cannot be used to connect to the external IP addresses of an instance.

To solve this issue follow steps below:

  1. add the SPARK_LOCAL_IP="<IP address>" to your configuration file as it suggested in console messages where IP address is local IP of your VM
  2. set network tag to your VM
  3. create firewall rule to enable incoming connections to your VM at port 4040
  4. check your firewall by running nmap -Pn EXTENAL_IP_OF_YOUR_VM from your pc
  5. check Web UI via browser http://EXTENAL_IP_OF_YOUR_VM:4040
user_124287940
Related