Hadoop YARN - Job consuming more resources than user limit factor

user_1305583 0
Viewed 150

I've got a default queue in YARN with the following configuration:

Capacity:   70%
Max Capacity:   90%
User Limit Factor   0.08
Minimum User Limit  8%
Maximum Applications    Inherited
Maximum AM Resource Inherited
Priority    0
Ordering Policy Fair
Maximum Application Lifetime    -1
Default Application Lifetime    -1
Enable Size Based Weight Ordering   Disabled
Maximum Allocation Vcores   Inherited
Maximum Allocation Mb   Inherited

But even with the User Limit Factor of 0.08 and Minimum User Limit of 8%, there are jobs running with more than 8% of the resources of the queue, as you can see below:

enter image description here

How is this even possible? Is the User Limit Factor/Minimum User Limit not working? Is there any other configuration I should be aware of?

hadoop-yarn hadoop
1 Answers

Minimum user limit of 8% means if 12 users arrive at the same time - they will get 8 % of resources each so as to give them equal share. However in your case the timing of the jobs start suggest that the tasks submitted first will have more cluster resources available since they are not coming in at the same time. A good understanding on these parameters can be had from here and here

enter image description here

user_14430840
Related