deploy Huggingface model to SageMaker endpoint

Question

I need to deploy a large language model (t0pp) on a SageMaker endpoint. I modified the official example to look like this:

from sagemaker.huggingface import HuggingFaceModel
import sagemaker

role = sagemaker.get_execution_role()

hub = {
  'HF_MODEL_ID':'bigscience/T0', # model_id from hf.co/models
  'HF_TASK':'text2text-generation' # NLP task you want to use for predictions
}

# create Hugging Face Model Class
huggingface_model = HuggingFaceModel(
   env=hub,
   role=role, # iam role with permissions to create an Endpoint
   transformers_version="4.6", # transformers version used
   pytorch_version="1.7", # pytorch version used
   py_version="py36", # python version of the DLC
)

# deploy model to SageMaker Inference
predictor = huggingface_model.deploy(
   initial_instance_count=1,
   instance_type="ml.m5.xlarge"
)

but I'm getting this error

UnexpectedStatusException: Error hosting endpoint huggingface-pytorch-inference-2022-09-21-15-44-30-116: Failed. Reason: The primary container for production variant AllTraffic did not pass the ping health check.

Any idea what is going wrong here?

deploy Huggingface model to SageMaker endpoint

0 Answers