`Table not initialized` when predicting with AI-platform

Viewed 133

I'm trying to save faster R-CNN hub model and use it with AI-platform gcloud ai-platform local predict. The error I'm getting is:

Failed to run the provided model: Exception during running the graph: [_Derived_]  Table not initialized.\n\t [[{{node hub_input/index_to_string_1_Lookup}}]]\n\t [[StatefulPartitionedCall_1/StatefulPartitionedCall/model/keras_layer/StatefulPartitionedCall]] (Error code: 2)\n'

The code for saving the model:

model_url = "https://tfhub.dev/google/faster_rcnn/openimages_v4/inception_resnet_v2/1"

input = tf.keras.Input(shape=(), dtype=tf.string)
decoded = tf.keras.layers.Lambda(
    lambda y: tf.map_fn(
        lambda x: tf.image.resize(
            tf.image.convert_image_dtype(
                tf.image.decode_jpeg(x, channels=3), tf.float32), (416, 416)
        ),
        tf.io.decode_base64(y), dtype=tf.float32)
)(input)

results = hub.KerasLayer(model_url, signature_outputs_as_dict=True)(decoded)

model = tf.keras.Model(inputs=input, outputs=results)
model.save("./saved_model", save_format="tf")

The model works when I load with tf.keras.models.load_model("./saved_model") and predict with it, but not with ai-platform local predict.

Command for ai-platform local predictions:

gcloud ai-platform local predict --model-dir ./saved_model --json-instances data.json --framework TENSORFLOW

Versions:

python 3.7.0
tensorflow==2.2.0
tensorflow-hub==0.7.0

Output of saved_model_cli:

MetaGraphDef with tag-set: 'serve' contains the following SignatureDefs:

signature_def['__saved_model_init_op']:
  The given SavedModel SignatureDef contains the following input(s):
  The given SavedModel SignatureDef contains the following output(s):
    outputs['__saved_model_init_op'] tensor_info:
        dtype: DT_INVALID
        shape: unknown_rank
        name: NoOp
  Method name is: 

signature_def['serving_default']:
  The given SavedModel SignatureDef contains the following input(s):
    inputs['image_bytes'] tensor_info:
        dtype: DT_STRING
        shape: (-1)
        name: serving_default_image_bytes:0
  The given SavedModel SignatureDef contains the following output(s):
    outputs['keras_layer'] tensor_info:
        dtype: DT_FLOAT
        shape: (-1, 4)
        name: StatefulPartitionedCall_1:0
    outputs['keras_layer_1'] tensor_info:
        dtype: DT_STRING
        shape: (-1, 1)
        name: StatefulPartitionedCall_1:1
    outputs['keras_layer_2'] tensor_info:
        dtype: DT_INT64
        shape: (-1, 1)
        name: StatefulPartitionedCall_1:2
    outputs['keras_layer_3'] tensor_info:
        dtype: DT_STRING
        shape: (-1, 1)
        name: StatefulPartitionedCall_1:3
    outputs['keras_layer_4'] tensor_info:
        dtype: DT_FLOAT
        shape: (-1, 1)
        name: StatefulPartitionedCall_1:4
  Method name is: tensorflow/serving/predict

Any ideas on how to fix the error?

1 Answers

The problem is that your input is being interpreted as a scalar. Do:

input = tf.keras.Input(shape=(1,), dtype=tf.string)
Related