Eclipse remote debug spark-submit

user_1479046 0
Viewed 4720

I want to do a remote debug of an application which is submitted to Spark using the Command Prompt in Windows.

How could I provide Spark the remote debug port to start.

apache-spark
2 Answers

you can always do the Remote Debugging using Spark with Java/scala. If you want to check with small dataset with localmode then you can use the step debugger. Have a look on the Java Debugger and how it works first.

Java step debugger

if you want to use the Spark specific Debugging in Remotemode Then check this link it has very nice example with step by step process.

spark-remote-debugging

For debugging the driver you can add the following to your spark-submit command. Then set your remote debugger to connect to the node you launched your driver program on.

--driver-java-options -agentlib:jdwp=transport=dt_socket,server=y,suspend=y,address=5005

You can customize the port according to your need. In this case, start your debugger in listening mode, then start your spark program and wait for the executor to attach to your debugger. It's important to set the number of executors to 1 or multiple executors will all try to connect to your debugger, likely causing problems. check the above example for more details.

If you are using jetbrains then you can use this example Spark remote debugging

user_22287980

Steps:

  1. export options, address below is the port to listen from IDE (Intellij or Eclipse)

export SPARK_SUBMIT_OPTS=-agentlib:jdwp=transport=dt_socket,server=y,suspend=y,address=8086

  1. Run spark application

    spark-submit --class prueba.SampleRDDTest pruebaTests-1.0-SNAPSHOT.jar

    Listening for transport dt_socket at address: 8086

  2. In IDE listen the machine where run the spark application, (i used a docker machine)

    Arguments to JVM: -agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=8086 host: 192.168.99.100 port: 8086

user_60416850
Related