Sagemaker tensorflow endpoint not calling the input_handler when being invoked for a prediction

user_3740545 0
Viewed 474

I'm deploying a tensorflow.serving endpoint with a custom inference.py script via the entry point parameter

model = Model(role='xxx',
              framework_version='2.2.0',
              entry_point='inference.py',
              model_data='xxx')

predictor = model.deploy(instance_type='xxx',
                         initial_instance_count=1,
                         endpoint_name='xxx')

inference.py constains an input_handler and an output_handler functions, but when i call predict with:

model = Predictor(endpoint_name='xxx')
url = 'xxx'

input = {
    'instances': [url]
}

predictions = model.predict(input)

I'm getting the following error:

botocore.errorfactory.ModelError: An error occurred (ModelError) when calling the InvokeEndpoint operation: Received client error (400) from model with message "{"error": "Failed to process element: 0 of 'instances' list. Error: Invalid argument: JSON Value: "xxx" Type: String is not of expected type: float" }"

It seems the function is never calling the input_handler function in inference.py script. Do you know why this might be happening?

amazon-sagemaker tensorflow
2 Answers

I am adding another possible cause for this error message, since it took me some time to resolve this.

I was using different sagemaker api versions (1.x and 2.x).

The name of the handler has changed from input_fn() to input_handler() for newer sagemaker tf containers.

Therefore the input_fn() was never called and the special input type never handled.

For details see: https://sagemaker.readthedocs.io/en/stable/frameworks/tensorflow/upgrade_from_legacy.html

Maybe this helps someone.

user_1849250

Found the problem thanks to AWS support:

I was creating an endpoint that already had an endpoint configuration with the same name and the new configuration wasn't being utilized.

user_37405450
Related