AWS Sagemaker suddenly and inconsistently throws resource limit exceeded error (limit 0), service quota says 10?

user_4458718 0
Viewed 377

I am suddenly getting the resource limit exceeded error

The account-level service limit 'ml.g4dn.xlarge for transform job usage' is 0 Instances, with current utilization of 0 Instances and a request delta of 1 Instances

However, I have been able to create training and batch jobs with this instance just fine before. When I check Service Quotas, under "Running Dedicated g4dn Hosts" the applied quota value is 1, but there is a closed quota increase request of 10. So why on earth would it say the limit is 0?

EDIT: I have also confirmed that it sometimes throws the error and sometimes fails. I.e. I had a ml.g4dn.2xlarge training job that ran successfully, then throws a 0 instance resource limit error shortly after. And if I try to clone the successful job it throws a resource 0 error.

amazon-sagemaker amazon-web-services
2 Answers

That's weird. The error you get

The account-level service limit 'ml.g4dn.xlarge for transform job usage' is 0 Instances, with current utilization of 0 Instances and a request delta of 1 Instances

Clearly means that your quota for instance ml.g4dn.xlarge is 0. You can also verify from here that 0 is the default value for that instance type for transform job, so unless you requested a quota increase it should be 0. You also say you have an increase quota ticket "closed", I have one too for quota increase and mine is "resolved", are you sure they accepted and not declined your request? Another important thing to check taht you are in the right region, quotas are for regions.

user_42674390

Default limits for ml.g4dn, ml.p3 and ml.p2 instances were recently set to 0, essentially meaning these instances are now by request only.

If you had a previous quota increase of 10 then that should still apply--are you sure the request wasn't denied?

In any case, you'll need to request a limit increase for that instance type to use it.

user_8956150
Related