When I get a prediction from sagemaker endpoint, what does the endpoint do?

user_12705481 0
Viewed 405

In sagemaker, the docs talk about inference scripts requiring to have 4 specific functions. When we get a prediction, the python SDK sends a request to the endpoint.

Then the inference script runs. But I cannot find where in the SDK the inference script is run.

When I navigate through the sdk code the Predictor.predict() method calls the sagemaker session to post a request to the endpoint and get a response. That is the final step in the sdk. Sagemaker is obviously doing something when it receives that request.

What is the code that it runs?

amazon-sagemaker
1 Answers

The endpoint is essentially a Flask web server running in a Docker container

If it's a scikit-learn image, when you invoke the endpoint, it loads your script from S3, then...

It calls input_fn(request_body: bytearray, content_type) -> np.ndarray to parse the request_body into a numpy array

Then it calls your model_fn(model_dir: str) -> object function to load the model from model_dir and return the model

Then it calls predict_fn(input_object: np.ndarray, model: object) -> np.array, which calls your model.predict() function and returns the prediction

Then it calls output_fn(prediction: np.array, accept: str) to take the result from predict_fn and encode it to the accept type

You don't need to implement all of these functions yourself, as there are defaults

You do need to implement model_fn

You only need to implement input_fn if you have non numeric data

You only need to implement predict_fn if your model uses something other than .predict()

You can see how the default function implementations work here

user_2234780
Related