new poster here. I've downloaded the SSD MobileNet v2 320x320 from https://github.com/tensorflow/models/blob/master/research/object_detection/g3doc/tf2_detection_zoo.md I have hosted it with a TensorFlow Serving Docker instance using the below command:
docker run -p 8500:8500 --name od --mount type=bind,source=C:\Users\johnno\Documents\Python\ObjectDetection\Models\ssd_mobilenet_v2_320x320_coco17_tpu-8\saved_model,target=/models/od -e MODEL_NAME=od -v=1 -t tensorflow/serving:latest
Which appears to start up alright. Note that I am intentionally NOT using the GPU version because once I have finished development I intend to host on a Raspberry Pi 4. (I will later have to use a lite model and quantize it). But for principality, I just wanted to get an SSD model hosted and inferencing via Python gRPC calls on my laptop PC.
Container Output below:
2022-09-05 21:35:53.230101: I external/org_tensorflow/tensorflow/core/util/util.cc:169] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
2022-09-05 21:35:53.236017: I tensorflow_serving/model_servers/server.cc:89] Building single TensorFlow model file config: model_name: od model_base_path: /models/od
2022-09-05 21:35:53.236610: I tensorflow_serving/model_servers/server_core.cc:465] Adding/updating models.
2022-09-05 21:35:53.236710: I tensorflow_serving/model_servers/server_core.cc:594] (Re-)adding model: od
2022-09-05 21:35:53.407370: I tensorflow_serving/core/basic_manager.cc:740] Successfully reserved resources to load servable {name: od version: 1}
2022-09-05 21:35:53.407413: I tensorflow_serving/core/loader_harness.cc:66] Approving load for servable version {name: od version: 1}
2022-09-05 21:35:53.407422: I tensorflow_serving/core/loader_harness.cc:74] Loading servable version {name: od version: 1}
2022-09-05 21:35:53.410889: I external/org_tensorflow/tensorflow/cc/saved_model/reader.cc:43] Reading SavedModel from: /models/od/1
2022-09-05 21:35:53.517275: I external/org_tensorflow/tensorflow/cc/saved_model/reader.cc:81] Reading meta graph with tags { serve }
2022-09-05 21:35:53.517340: I external/org_tensorflow/tensorflow/cc/saved_model/reader.cc:122] Reading SavedModel debug info (if present) from: /models/od/1
2022-09-05 21:35:53.519963: I external/org_tensorflow/tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 AVX512F AVX512_VNNI FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2022-09-05 21:35:53.695782: I external/org_tensorflow/tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:354] MLIR V1 optimization pass is not enabled
2022-09-05 21:35:53.761113: I external/org_tensorflow/tensorflow/cc/saved_model/loader.cc:228] Restoring SavedModel bundle.
2022-09-05 21:35:54.778557: I external/org_tensorflow/tensorflow/cc/saved_model/loader.cc:212] Running initialization op on SavedModel bundle at path: /models/od/1
2022-09-05 21:35:55.538255: I external/org_tensorflow/tensorflow/cc/saved_model/loader.cc:301] SavedModel load for tags { serve }; Status: success: OK. Took 2127366 microseconds.
2022-09-05 21:35:55.586564: I tensorflow_serving/servables/tensorflow/saved_model_warmup_util.cc:59] No warmup data file found at /models/od/1/assets.extra/tf_serving_warmup_requests
2022-09-05 21:35:55.647356: I tensorflow_serving/core/loader_harness.cc:95] Successfully loaded servable version {name: od version: 1}
2022-09-05 21:35:55.650036: I tensorflow_serving/model_servers/server_core.cc:486] Finished adding/updating models
2022-09-05 21:35:55.650096: I tensorflow_serving/model_servers/server.cc:133] Using InsecureServerCredentials
2022-09-05 21:35:55.650117: I tensorflow_serving/model_servers/server.cc:395] Profiler service is enabled
2022-09-05 21:35:55.651523: I tensorflow_serving/model_servers/server.cc:421] Running gRPC ModelServer at 0.0.0.0:8500 ...
[warn] getaddrinfo: address family for nodename not supported
2022-09-05 21:35:55.657032: I tensorflow_serving/model_servers/server.cc:442] Exporting HTTP/REST API at:localhost:8501 ...
[evhttp_server.cc : 245] NET_LOG: Entering the event loop ...
To me this output looks alright, and as you see it enters the event loop listening on gRPC port 8500.
Now when I attempt to inference with Python using the following code, I never get a response and the request always times out. I increased the timeout threshold from 60 seconds to 20 minutes to check it's not just taking ages... but alas it still exceeds timeout.
from werkzeug.utils import secure_filename
import numpy as np
import os
import sys
import tensorflow as tf
from PIL import Image, ImageOps
import time
import cv2
import grpc
from grpc.beta import implementations
sys.path.append('..')
from object_detection.utils import label_map_util
from object_detection.utils import visualization_utils as viz_utils
from tensorflow_serving.apis import predict_pb2
from tensorflow_serving.apis import prediction_service_pb2, get_model_metadata_pb2
from tensorflow_serving.apis import prediction_service_pb2_grpc
from object_detection.core.standard_fields import DetectionResultFields as dt_fields
%matplotlib inline
tf.get_logger().setLevel('ERROR')
PATH_TO_LABELS = './ssd_mobilenet_v2_fpnlite_3200x320_coco17_tpu-8/data/mscoco_label_map.pbtxt'
NUM_CLASSES = 90
label_map = label_map_util.load_labelmap(PATH_TO_LABELS)
categories = label_map_util.convert_label_map_to_categories(label_map, max_num_classes=NUM_CLASSES, use_display_name=True)
category_index = label_map_util.create_category_index(categories)
# configure grpc channel
def get_stub(host='127.0.0.1', port='8500'):
channel = grpc.insecure_channel('127.0.0.1:8500')
stub = prediction_service_pb2_grpc.PredictionServiceStub(channel)
return stub
def load_image_into_numpy_array(image):
(im_width, im_height) = image.size
return np.array(image.getdata()).reshape((im_height, im_width,3)).astype(np.uint8)
def load_input_tensor(input_image):
image_np = load_image_into_numpy_array(input_image)
image_np_expanded = np.expand_dims(image_np, axis=0).astype(np.uint8)
tensor = tf.make_tensor_proto(image_np_expanded)
return tensor
def inference(frame, stub, model_name='od'):
channel = grpc.insecure_channel('localhost:8500', options=(('grpc.enable_http_proxy', 0),))
print('Channel: ', channel)
stub = prediction_service_pb2_grpc.PredictionServiceStub(channel)
print('Stub: ', stub)
request = predict_pb2.PredictRequest()
request.model_spec.name = 'od'
#request.model_spec.signature_name = 'serving_default'
cv2_im = cv2.cvtColor(frame,cv2.COLOR_BGR2RGB)
image = Image.fromarray(cv2_im)
input_tensor = load_input_tensor(image)
request.inputs['input_tensor'].CopyFrom(input_tensor)
#print('Request:', request)
result = stub.Predict(request, 60.0)
image_np = load_image_into_numpy_array(image)
output_dict = {}
output_dict['detection_classes'] = np.squeeze(result.outputs[dt_fields.detection_classes].float_val).astype(np.uint8)
output_dict['detection_boxes'] = np.reshape(result.outputs[dt_fields.detection_boxes].float_val, (-1,4))
output_dict['detection_scores'] = np.squeeze(result.outputs[dt_fields.detection_scores].float_val)
# Convert from RGB back to BGR before returning image
image_np = cv2.cvtColor(image_np,cv2.COLOR_RGB2BGR)
frame = viz_utils.visualize_boxes_and_labels_on_image_array(image_np,
output_dict['detection_boxes'],
output_dict['detection_classes'],
output_dict['detection_scores'],
category_index,
use_normalized_coordinates=True,
max_boxes_to_draw=200,
min_score_thresh=0.30,
agnostic_mode=False)
return (frame)
PATH_TO_TEST_IMAGES_DIR = './images/'
filename = 'walking.jpg'
TEST_IMAGE_PATHS = [os.path.join(PATH_TO_TEST_IMAGES_DIR, filename)]
stub = get_stub()
for image_path in TEST_IMAGE_PATHS:
image_np = Image.open(image_path)
image_np = ImageOps.fit(image_np, (320,320), Image.ANTIALIAS)
image_np = np.array(image_np)
print(image_np.shape)
image_np_inferenced = inference(image_np, stub)
im = Image.fromarray(image_np_inferenced)
im.save('./inferenced/' + filename)
The output I receive is this:
(320, 320, 3)
Channel: <grpc._channel.Channel object at 0x0000024A20D69EA0>
Stub: <tensorflow_serving.apis.prediction_service_pb2_grpc.PredictionServiceStub object at 0x0000024A20D6A200>
---------------------------------------------------------------------------
_InactiveRpcError Traceback (most recent call last)
Input In [78], in <cell line: 5>()
8 image_np = np.array(image_np)
9 print(image_np.shape)
---> 10 image_np_inferenced = inference(image_np, stub)
11 im = Image.fromarray(image_np_inferenced)
12 im.save('./inferenced/' + filename)
Input In [77], in inference(frame, stub, model_name)
12 request.inputs['input_tensor'].CopyFrom(input_tensor)
13 #print('Request:', request)
---> 14 result = stub.Predict(request, 60.0)
15 image_np = load_image_into_numpy_array(image)
17 output_dict = {}
File ~\AppData\Local\Programs\Python\Python310\lib\site-packages\grpc\_channel.py:946, in _UnaryUnaryMultiCallable.__call__(self, request, timeout, metadata, credentials, wait_for_ready, compression)
937 def __call__(self,
938 request,
939 timeout=None,
(...)
942 wait_for_ready=None,
943 compression=None):
944 state, call, = self._blocking(request, timeout, metadata, credentials,
945 wait_for_ready, compression)
--> 946 return _end_unary_response_blocking(state, call, False, None)
File ~\AppData\Local\Programs\Python\Python310\lib\site-packages\grpc\_channel.py:849, in _end_unary_response_blocking(state, call, with_call, deadline)
847 return state.response
848 else:
--> 849 raise _InactiveRpcError(state)
_InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
status = StatusCode.DEADLINE_EXCEEDED
details = "Deadline Exceeded"
debug_error_string = "UNKNOWN:Error received from peer ipv6:%5B::1%5D:8500 {grpc_message:"Deadline Exceeded", grpc_status:4, created_time:"2022-09-05T21:39:41.387573683+00:00"}"
>
I apologise for my messy code. This has previously worked for me with the Faster RCNN type models on a GPU. Should I have changed something to allow CPU inference, or should it still work regardless but just be slower? For reference I am using:
- tensorflow - 2.9.2
- tensorflow-serving-api - 2.9.1
- grpcio - 1.49.0rc3
- pip - 22.2.2
- Python - 3.10.6
Anyone got any idea what I'm doing wrong here? Any help would be appreciated. Many thanks.