Amazon Sagemaker ResourceLimitExceeded Error for XGBoost (Free Tier)

user_6520041 0
Viewed 5276

I am trying to create an XGBoost model in free tier AWS Sagemaker. I am getting an error of:

"ResourceLimitExceeded: An error occurred (ResourceLimitExceeded) when calling the CreateEndpoint operation: The account-level service limit 'ml.m5.xlarge for endpoint usage' is 0 Instances, with current utilization of 0 Instances and a request delta of 1 Instances.".

What is the right train_instance_type I should use?

Here is my code:

# import libraries
import boto3, re, sys, math, json, os, sagemaker, urllib.request
from sagemaker import get_execution_role
import numpy as np                                
import pandas as pd                               
import matplotlib.pyplot as plt                   
from IPython.display import Image                 
from IPython.display import display               
from time import gmtime, strftime                 
from sagemaker.predictor import csv_serializer   

# Define IAM role
role = get_execution_role()
prefix = 'sagemaker/DEMO-xgboost-dm'
containers = {'us-west-2': '433757028032.dkr.ecr.us-west-2.amazonaws.com/xgboost:latest',
              'us-east-1': '811284229777.dkr.ecr.us-east-1.amazonaws.com/xgboost:latest',
              'us-east-2': '825641698319.dkr.ecr.us-east-2.amazonaws.com/xgboost:latest',
              'eu-west-1': '685385470294.dkr.ecr.eu-west-1.amazonaws.com/xgboost:latest'} # each region has its XGBoost container
my_region = boto3.session.Session().region_name # set the region of the instance

# Create an instance of the XGBoost model (an estimator), and define the model’s hyperparameters.
# Note: train_instance_type='ml.m5.large' has 0 free credits! Use one of https://aws.amazon.com/sagemaker/pricing/ 
sess = sagemaker.Session()
xgb = sagemaker.estimator.Estimator(containers[my_region],role, train_instance_count=1, train_instance_type='ml.m5.xlarge',output_path='s3://{}/{}/output'.format('my_s3_bucket', prefix),sagemaker_session=sess)
xgb.set_hyperparameters(max_depth=1,eta=0.2,gamma=4,min_child_weight=6,subsample=0.8,silent=0,objective='binary:logistic',num_round=100)
# Train the model using gradient optimization on a ml.m4.xlarge instance
# After a few minutes, you should start to see the training logs being generated.
xgb.fit({'train': s3_input_train})

At this step this is what I see:

2019-10-22 06:32:51 Starting - Starting the training job...
2019-10-22 06:33:00 Starting - Launching requested ML instances......
2019-10-22 06:33:54 Starting - Preparing the instances for training...
2019-10-22 06:34:41 Downloading - Downloading input data...
2019-10-22 06:35:22 Training - Training image download completed. Training in progress..Arguments: train
[2019-10-22:06:35:22:INFO] Running standalone xgboost training.
[2019-10-22:06:35:22:INFO] Path /opt/ml/input/data/validation does not exist!
[2019-10-22:06:35:22:INFO] File size need to be processed in the node: 3.38mb. Available memory size in the node: 8089.9mb
[2019-10-22:06:35:22:INFO] Determined delimiter of CSV input is ','
[06:35:22] S3DistributionType set as FullyReplicated
[06:35:22] 28831x59 matrix with 1701029 entries loaded from /opt/ml/input/data/train?format=csv&label_column=0&delimiter=,
[06:35:22] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 2 extra nodes, 0 pruned nodes, max_depth=1
[0]#011train-error:0.102182
[06:35:22] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 2 extra nodes, 0 pruned nodes, max_depth=1
[1]#011train-error:0.102182
[06:35:22] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 2 extra nodes, 0 pruned nodes, max_depth=1
[2]#011train-error:0.102182
[06:35:22] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 2 extra nodes, 0 pruned nodes, max_depth=1
[3]#011train-error:0.102182
[06:35:22] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 2 extra nodes, 0 pruned nodes, max_depth=1
[4]#011train-error:0.102182
[06:35:22] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 2 extra nodes, 0 pruned nodes, max_depth=1
[5]#011train-error:0.102182
[06:35:22] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 2 extra nodes, 0 pruned nodes, max_depth=1
[6]#011train-error:0.102182
[06:35:22] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 2 extra nodes, 0 pruned nodes, max_depth=1
[7]#011train-error:0.10839
[06:35:22] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 2 extra nodes, 0 pruned nodes, max_depth=1
[8]#011train-error:0.102737
[06:35:22] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 2 extra nodes, 0 pruned nodes, max_depth=1
[9]#011train-error:0.107697

And then when I deploy this:

# Deploy the model on a server and create an endpoint that you can access
xgb_predictor = xgb.deploy(initial_instance_count=1,instance_type='ml.m5.xlarge')
---------------------------------------------------------------------------
ResourceLimitExceeded                     Traceback (most recent call last)
<ipython-input-38-6d149f3edc98> in <module>()
      1 # Deploy the model on a server and create an endpoint that you can access
----> 2 xgb_predictor = xgb.deploy(initial_instance_count=1,instance_type='ml.m5.xlarge')

~/anaconda3/envs/python3/lib/python3.6/site-packages/sagemaker/estimator.py in deploy(self, initial_instance_count, instance_type, accelerator_type, endpoint_name, use_compiled_model, update_endpoint, wait, model_name, kms_key, **kwargs)
    559             tags=self.tags,
    560             wait=wait,
--> 561             kms_key=kms_key,
    562         )
    563 

~/anaconda3/envs/python3/lib/python3.6/site-packages/sagemaker/model.py in deploy(self, initial_instance_count, instance_type, accelerator_type, endpoint_name, update_endpoint, tags, kms_key, wait)
    464         else:
    465             self.sagemaker_session.endpoint_from_production_variants(
--> 466                 self.endpoint_name, [production_variant], tags, kms_key, wait
    467             )
    468 

~/anaconda3/envs/python3/lib/python3.6/site-packages/sagemaker/session.py in endpoint_from_production_variants(self, name, production_variants, tags, kms_key, wait)
   1361 
   1362             self.sagemaker_client.create_endpoint_config(**config_options)
-> 1363         return self.create_endpoint(endpoint_name=name, config_name=name, tags=tags, wait=wait)
   1364 
   1365     def expand_role(self, role):

~/anaconda3/envs/python3/lib/python3.6/site-packages/sagemaker/session.py in create_endpoint(self, endpoint_name, config_name, tags, wait)
    975 
    976         self.sagemaker_client.create_endpoint(
--> 977             EndpointName=endpoint_name, EndpointConfigName=config_name, Tags=tags
    978         )
    979         if wait:

~/anaconda3/envs/python3/lib/python3.6/site-packages/botocore/client.py in _api_call(self, *args, **kwargs)
    355                     "%s() only accepts keyword arguments." % py_operation_name)
    356             # The "self" in this scope is referring to the BaseClient.
--> 357             return self._make_api_call(operation_name, kwargs)
    358 
    359         _api_call.__name__ = str(py_operation_name)

~/anaconda3/envs/python3/lib/python3.6/site-packages/botocore/client.py in _make_api_call(self, operation_name, api_params)
    659             error_code = parsed_response.get("Error", {}).get("Code")
    660             error_class = self.exceptions.from_code(error_code)
--> 661             raise error_class(parsed_response, operation_name)
    662         else:
    663             return parsed_response

ResourceLimitExceeded: An error occurred (ResourceLimitExceeded) when calling the CreateEndpoint operation: The account-level service limit 'ml.m5.xlarge for endpoint usage' is 0 Instances, with current utilization of 0 Instances and a request delta of 1 Instances. Please contact AWS support to request an increase for this limit.

Edit: Trying ml.m4.xlarge instance:

When I use ml.m4.xlarge, I get the same message of "ResourceLimitExceeded: An error occurred (ResourceLimitExceeded) when calling the CreateEndpoint operation: The account-level service limit 'ml.m4.xlarge for endpoint usage' is 0 Instances, with current utilization of 0 Instances and a request delta of 1 Instances. Please contact AWS support to request an increase for this limit."

amazon-sagemaker boto3 amazon-web-services python
3 Answers

Steps to Request Limit Increase for ml.m5.xlarge

  1. visit aws console https://console.aws.amazon.com/
  2. click on support on the top right corner
  3. click create a case (orange button)
  4. select Service Limit Increase radio button
  5. For Limit Type, Search and Select SageMaker Notebook Instances
  6. select the same region as the region that is displayed on the top right corner of your amazon console.
  7. Write a short Use case description
  8. For Limit, Select ml.[x].[x] (in your case, ml.m5.xlarge)
  9. New Limit Values 1

It may take 48 hours for this manual support ticket to turn around.(For me I got a reply from support team after one day and the instance limit is changed to 1)

user_122347530

According to this AWS page you are getting 50 hours per month of m4.xlarge for training for the first two months and 125 hours per month of m4.xlarge for hosting for the first two months. So if you are within the first two months the ml.m4.xlarge should do the trick.

As for service limit itself according to this post newly created account has every instance type (other than t2 medium) in SageMaker limited to 0, instead of default limit.

So you need to contact AWS support after all and ask to increase your limit. Also, this might be limited by the admin of your account if you not an admin yourself. So, in that case, this should be your first port of call.

user_110787250

Based on your provided output, the model is training successfully. What is failing is the deploy step which hosts your model as a queryable endpoint. Limits on Inference are separate from those of Training. According to this page SageMaker currently provides

125 hours of m4.xlarge or m5.xlarge instance per month on Inference

if you are within the first 2 months of activation.

You can check your current SageMaker limits using the Service Quotas console or API. If you currently do not have any allocation of the above instance types for the Endpoint resource, you can request a limit increase through Service Quotas as well. More details here.

user_82072290
Related