I'm stuck in a situation where Sagemaker is looking for a docker image in ECS registry which I had to remove and I can't figure out how to make it forget about that.
I had to rebuild a docker with torchserve for Sagemaker. I removed the old one (let's call it torchserve-old-name, and uploaded a new torchserve:v2. For some weird reason SM is still looking for the old one.
from sagemaker.model import Model
from sagemaker.predictor import Predictor
image = "134244564256.dkr.ecr.us-west-2.amazonaws.com/torchserve:v2"
model_data = "s3://my-models/torchserve/my-model.tar.gz"
sm_model_name = 'my-model'
print(f"docker image: {image}")
# docker image: 134244564256.dkr.ecr.us-west-2.amazonaws.com/torchserve:v2
torchserve_model = Model(model_data = model_data,
image_uri = image,
role = role,
predictor_cls=Predictor,
name = sm_model_name)
endpoint_name = 'torchserve-endpoint-' + time.strftime("%Y-%m-%d-%H-%M-%S", time.gmtime())
predictor = torchserve_model.deploy(instance_type='ml.m4.xlarge',
initial_instance_count=1,
endpoint_name = endpoint_name)
Results in:
---------------------------------------------------------------------------
UnexpectedStatusException Traceback (most recent call last)
<ipython-input-12-67fff0e32d2d> in <module>
3 predictor = torchserve_model.deploy(instance_type='ml.m4.xlarge',
4 initial_instance_count=1,
----> 5 endpoint_name = endpoint_name)
~/anaconda3/envs/python3/lib/python3.6/site-packages/sagemaker/model.py in deploy(self, initial_instance_count, instance_type, serializer, deserializer, accelerator_type, endpoint_name, tags, kms_key, wait, data_capture_config, **kwargs)
762 kms_key=kms_key,
763 wait=wait,
--> 764 data_capture_config_dict=data_capture_config_dict,
765 )
766
~/anaconda3/envs/python3/lib/python3.6/site-packages/sagemaker/session.py in endpoint_from_production_variants(self, name, production_variants, tags, kms_key, wait, data_capture_config_dict)
3454
3455 self.sagemaker_client.create_endpoint_config(**config_options)
-> 3456 return self.create_endpoint(endpoint_name=name, config_name=name, tags=tags, wait=wait)
3457
3458 def expand_role(self, role):
~/anaconda3/envs/python3/lib/python3.6/site-packages/sagemaker/session.py in create_endpoint(self, endpoint_name, config_name, tags, wait)
2957 )
2958 if wait:
-> 2959 self.wait_for_endpoint(endpoint_name)
2960 return endpoint_name
2961
~/anaconda3/envs/python3/lib/python3.6/site-packages/sagemaker/session.py in wait_for_endpoint(self, endpoint, poll)
3243 ),
3244 allowed_statuses=["InService"],
-> 3245 actual_status=status,
3246 )
3247 return desc
UnexpectedStatusException: Error hosting endpoint torchserve-endpoint-2021-02-06-21-39-31: Failed. Reason:
The repository 'torchserve-old-name' does not exist in the registry with id '134244564256'..
I am 100% sure that the old image 'torchserve-old-name' has not been mentioned in this code. I've restarted and re-ran notebook.
Where is it cached, and how can I clear that cache? Sagemaker function documentation doesn't seem to mention it.