CloudEra - Spark - On Call Cycles Through All Spark UI Ports, Even Though Only 3 Applications are Currently Running

user_9831522 0
Viewed 51

For the simple example we currently have three running jobs on the box. Overtime I have seen on Cluster 1 many "Ghost Ports", where a port, lets say 4040 was able to be used for a long time, now is forever held by a ghost-process.

My cluster's boxes are within their own VLAN, with all ports open between them.

Call

spark-shell

Output

20/03/06 12:54:21 WARN util.Utils: Service 'SparkUI' could not bind on port 4040. Attempting port 4041.
20/03/06 12:54:21 WARN util.Utils: Service 'SparkUI' could not bind on port 4041. Attempting port 4042.
20/03/06 12:54:21 WARN util.Utils: Service 'SparkUI' could not bind on port 4042. Attempting port 4043.
20/03/06 12:54:21 WARN util.Utils: Service 'SparkUI' could not bind on port 4043. Attempting port 4044.
20/03/06 12:54:21 WARN util.Utils: Service 'SparkUI' could not bind on port 4044. Attempting port 4045.
20/03/06 12:54:21 WARN util.Utils: Service 'SparkUI' could not bind on port 4045. Attempting port 4046.
20/03/06 12:54:21 WARN util.Utils: Service 'SparkUI' could not bind on port 4046. Attempting port 4047.
20/03/06 12:54:21 WARN util.Utils: Service 'SparkUI' could not bind on port 4047. Attempting port 4048.
20/03/06 12:54:21 WARN util.Utils: Service 'SparkUI' could not bind on port 4048. Attempting port 4049.
20/03/06 12:54:21 WARN util.Utils: Service 'SparkUI' could not bind on port 4049. Attempting port 4050.
20/03/06 12:54:21 WARN util.Utils: Service 'SparkUI' could not bind on port 4050. Attempting port 4051.
20/03/06 12:54:21 WARN util.Utils: Service 'SparkUI' could not bind on port 4051. Attempting port 4052.
20/03/06 12:54:21 WARN util.Utils: Service 'SparkUI' could not bind on port 4052. Attempting port 4053.
20/03/06 12:54:21 WARN util.Utils: Service 'SparkUI' could not bind on port 4053. Attempting port 4054.
20/03/06 12:54:21 WARN util.Utils: Service 'SparkUI' could not bind on port 4054. Attempting port 4055.
20/03/06 12:54:21 WARN util.Utils: Service 'SparkUI' could not bind on port 4055. Attempting port 4056.

I have tried

  1. Reseting the box
  2. Find the jobs by searching for their PIDs by grepping Spark
  3. Using the 'yarn application' CLI commands
  4. Killing the cloudera server and agents on spark master/edge node.

Is there anything I can do to get these ports back?

cloudera hadoop-yarn apache-spark
1 Answers

Found a work-around to fix the issue. It seems that for our environment our Windows VDIs rotate and dont "totally" shut-down on the weekends. Because of this the Java side was keeping the actions open, but YARN was identifying that it closed, but no closing the java action.

My bad work-around was to run:

ps -aux | grep spark > jobs.log

If you do SPARK instead it will output bad results. When looking inside of the log I was able to find Spark pids open from January and February. When running the below command the ports opened up. Word to the wise, if using CloudEra dont exit your spark shells by CTRL+C you need to do CTRL+D.

kill -9 ${PID}

Leaving question unanswered for anyone with a better workaround.

user_98315220
Related