ModelNotReadyException when calling sagemaker

user_7656163 0
Viewed 52

Last night I received this error twice when invoking a serverless sagemaker endpoint

serverless ClientError: An error occurred (ModelNotReadyException) when calling the InvokeEndpoint operation (reached max retries: 4): Model for endpoint electra-nlp-2022-06-14-18-32-00-249 variant AllTraffic is not ready for inference yet.

The third time, and every time after I invoked the endpoint it worked. Does anyone know what could have caused this error. Although it works now, I can't be having random ModelNotReadyException's when the sagemaker endpoint is invoked.

amazon-sagemaker serverless
1 Answers

What are your CloudWatch logs showing for the failed invocations? Could you share/paste those here. You can configure/increase retries in your SDK client that is invoking the endpoint, adding a code sample below and please check out this link: https://aws.amazon.com/premiumsupport/knowledge-center/sagemaker-python-throttlingexception/.

import boto3 
from botocore.config import Config
sm_boto = boto3.client('sagemaker', config=Config(connect_timeout=5, read_timeout=60, retries={'max_attempts': 20}))
print(sm_boto.meta.config.retries)
user_165046400
Related