I am using the org.apache.spark.deploy.yarn.Client API in Java to submit Spark application to YARN.
SparkConf sparkConf = new SparkConf();
List<String> submitArgs = new ArrayList<>();
if (StringUtils.hasText(appName)) {
submitArgs.add("--name");
submitArgs.add(appName);
sparkConf.setAppName(appName);
}
submitArgs.add("--jar");
submitArgs.add(appJarPath);
submitArgs.add("--class");
submitArgs.add(appMainClass);
System.setProperty("SPARK_YARN_MODE", "true");
sparkConf.setMaster("yarn")
.set("spark.submit.deployMode", "cluster")
.set("spark.yarn.queue",queue);
ClientArguments clientArguments = new ClientArguments(submitArgs.toArray(new String[submitArgs.size()]));
Client client = new Client(clientArguments, sparkConf);
client.run();
I am trying to create a scenario when the client cannot connect to the resource manager, after so may attempts, it will timeout and throw back an exception. However, the client keeps retrying connecting to the YARN resource manager without any timeout. See below console output:
2018-09-18 14:20:25.046 INFO 10480 --- [nio-8080-exec-1] org.apache.hadoop.yarn.client.RMProxy : Connecting to ResourceManager at /0.0.0.0:8032
2018-09-18 14:20:27.600 INFO 10480 --- [nio-8080-exec-1] org.apache.hadoop.ipc.Client : Retrying connect to server: 0.0.0.0/0.0.0.0:8032. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2018-09-18 14:20:29.603 INFO 10480 --- [nio-8080-exec-1] org.apache.hadoop.ipc.Client : Retrying connect to server: 0.0.0.0/0.0.0.0:8032. Already tried 1 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2018-09-18 14:20:31.600 INFO 10480 --- [nio-8080-exec-1] org.apache.hadoop.ipc.Client : Retrying connect to server: 0.0.0.0/0.0.0.0:8032. Already tried 2 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2018-09-18 14:20:33.607 INFO 10480 --- [nio-8080-exec-1] org.apache.hadoop.ipc.Client : Retrying connect to server: 0.0.0.0/0.0.0.0:8032. Already tried 3 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2018-09-18 14:20:35.608 INFO 10480 --- [nio-8080-exec-1] org.apache.hadoop.ipc.Client : Retrying connect to server: 0.0.0.0/0.0.0.0:8032. Already tried 4 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2018-09-18 14:20:37.619 INFO 10480 --- [nio-8080-exec-1] org.apache.hadoop.ipc.Client : Retrying connect to server: 0.0.0.0/0.0.0.0:8032. Already tried 5 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2018-09-18 14:20:39.620 INFO 10480 --- [nio-8080-exec-1] org.apache.hadoop.ipc.Client : Retrying connect to server: 0.0.0.0/0.0.0.0:8032. Already tried 6 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2018-09-18 14:20:41.623 INFO 10480 --- [nio-8080-exec-1] org.apache.hadoop.ipc.Client : Retrying connect to server: 0.0.0.0/0.0.0.0:8032. Already tried 7 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2018-09-18 14:20:43.633 INFO 10480 --- [nio-8080-exec-1] org.apache.hadoop.ipc.Client : Retrying connect to server: 0.0.0.0/0.0.0.0:8032. Already tried 8 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2018-09-18 14:20:45.632 INFO 10480 --- [nio-8080-exec-1] org.apache.hadoop.ipc.Client : Retrying connect to server: 0.0.0.0/0.0.0.0:8032. Already tried 9 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2018-09-18 14:21:18.665 INFO 10480 --- [nio-8080-exec-1] org.apache.hadoop.ipc.Client : Retrying connect to server: 0.0.0.0/0.0.0.0:8032. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2018-09-18 14:21:20.676 INFO 10480 --- [nio-8080-exec-1] org.apache.hadoop.ipc.Client : Retrying connect to server: 0.0.0.0/0.0.0.0:8032. Already tried 1 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2018-09-18 14:21:22.679 INFO 10480 --- [nio-8080-exec-1] org.apache.hadoop.ipc.Client : Retrying connect to server: 0.0.0.0/0.0.0.0:8032. Already tried 2 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2018-09-18 14:21:24.686 INFO 10480 --- [nio-8080-exec-1] org.apache.hadoop.ipc.Client : Retrying connect to server: 0.0.0.0/0.0.0.0:8032. Already tried 3 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2018-09-18 14:21:26.691 INFO 10480 --- [nio-8080-exec-1] org.apache.hadoop.ipc.Client : Retrying connect to server: 0.0.0.0/0.0.0.0:8032. Already tried 4 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2018-09-18 14:21:28.695 INFO 10480 --- [nio-8080-exec-1] org.apache.hadoop.ipc.Client : Retrying connect to server: 0.0.0.0/0.0.0.0:8032. Already tried 5 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2018-09-18 14:21:30.697 INFO 10480 --- [nio-8080-exec-1] org.apache.hadoop.ipc.Client : Retrying connect to server: 0.0.0.0/0.0.0.0:8032. Already tried 6 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2018-09-18 14:21:32.708 INFO 10480 --- [nio-8080-exec-1] org.apache.hadoop.ipc.Client : Retrying connect to server: 0.0.0.0/0.0.0.0:8032. Already tried 7 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
...
What custom configuration I should do and how so that the retry will stop after a limited number of attempts or time?