My Cluster has 2 queues:
default - 50% and cannot go up.
important - 50% and can go beyond 50.
See my configuration below:
<configuration>
<property><name>yarn.scheduler.capacity.root.queues</name><value>default,important</value></property>
<property><name>yarn.scheduler.capacity.root.capacity</name><value>100</value></property>
<property><name>yarn.scheduler.capacity.root.default.capacity</name><value>50</value>
<property><name>yarn.scheduler.capacity.root.important.capacity</name><value>50</value>
<property><name>yarn.scheduler.capacity.root.default.maximum-capacity</name><value>50</value>
<property><name>yarn.scheduler.capacity.resource-calculator</name><value>org.apache.hadoop.yarn.util.resource.DominantResourceCalculator</value>
<property><name>yarn.scheduler.capacity.root.support.user-limit-factor</name><value>2</value>
</configuration>
The issue is that the important queue is not able to go beyond its dedicated capacity (especially vcores) when a single spark application is ran