I'm executing some spark(scala) sql code in spark shell. I want to know which queue I am using and if possible I want to know how much memory and executors I am using and how to optimize it?
3 Answers
You can set queue name, number of executors, executor memory, number of total cores, cores per executor, driver memory,etc when you start spark shell or spark-submit
here is how you can specify the parameters.
spark-shell --executor-memory 6G --executor-cores 5 --num-executors 20 --driver-memory 2G --queue $queue_name
You should be calculating these parameters as per your cluster capacity according to fat executor or thin executor concept.
If you still want to check resources utilization, you can check resource manager page or SPARK web UI page
You should check Resource Manager page for Yarn... All the jobs running on cluster will be listed on this page. Click on your job and in details of that job you will get queue information
You can control which queue to use while starting spark shell by command line option --queue. If you do not have access to submit jobs to provided queue then spark shell initialization will fail.
Similarly, you can specify other resources such number of executors, memory and cores for each executor on command line.
When you are submitting spark application using spark submit then that time also you provide same.
Related