Background:
In a vision application on a GCP, we are using TF serving. The application using TF Serving is written in Go. This application converts the image to Tensor and sends it to the TF serving using gRPC.
Problem:
The preprocessing logic in Golang does not work as well as it does in Python, using Keras image library (the accuracy of inference suffers). A part of the reason could be that Python libraries were used during training.
We tried,
Tensorflow serving provides a way to introduce a pre-processor that can run on the serving container. It seems to have limited functionality (can't package Keras library with the model). We tried the following two options
What works is Keras Preprocessing (Python), on the client side as follows.
img = tf.keras.preprocessing.image.load_img(file_name, target_size=(HEIGHT, WIDTH))
img_array = tf.keras.preprocessing.image.img_to_array(img)
… grpc call to TensorflowServing...
Our goal is to use “serving_input_receiver_fn” and preprocess image in the TFServing space as described in this blog post: https://medium.com/devseed/technical-walkthrough-packaging-ml-models-for-inference-with-tf-serving-2a50f73ce6f8
But the following code which is executed as “serving_input_receiver_fn” does not yield the correct inferences.
image = tf.image.decode_image(image_str_tensor, channels=CHANNELS dtype=tf.uint8)
image = tf.reshape(image, [HEIGHT, WIDTH, CHANNELS])
Our goal is to run the following Keras code (in a similar way) inside ““serving_input_receiver_fn” (assuming that we can load the image from “grpc” stream).
img = tf.keras.preprocessing.image.load_img(file_name, target_size=(HEIGHT, WIDTH))
img_array = tf.keras.preprocessing.image.img_to_array(img)
Is it possible? This is a massive deployment (70 GPUs & 2300 CPUs) therefore every bit of the performance counts. In our case, the image preprocessing on TF-Serving machine is the most optimal.