Where should pre-processing and post-processing steps be executed when a TF model is served using TensorFlow serving?

user_2543480 0
Viewed 5857

Typically to use a TF graph, it is necessary to convert raw data to numerical values. I refer to this process as a pre-processing step. For example, if the raw data is a sentence, one way is to do this is to tokenize the sentence and map each word to a unique number. This preprocessing creates a sequence of number for each sentence, which will be the input of the model.

We need also to post-process the output of a model to interpret it. For example, converting a sequence of numbers generated by the model to words and then building a sentence.

TF Serving is a new technology that is recently introduced by Google to serve a TF model. My question is that:

Where should pre-processing and post-processing be executed when a TF model is served using TensorFlow serving?

Should I encapsulate pre-processing and post-processing steps in my TF Graph (e.g. using py_fun or map_fn) or there is another TensorFlow technology that I am not aware of.

tensorflow-serving tensorflow
1 Answers

I'm running over the same issue here, even if I'm not 100% sure yet on how to use the wordDict variable (I guess you use one too to map the words with its ids), the main pre-process and post-process functions are defined here:

https://www.tensorflow.org/programmers_guide/saved_model

as export_outputs and serving_input_receiver_fn.

  • exports_outputs

Needs to be defined in EstimatorSpec if you are using estimators. Here is an example for a classification algorithm

  predicted_classes = tf.argmax(logits, 1)
  categories_tensor = tf.convert_to_tensor(CATEGORIES, tf.string)
  export_outputs = { "categories": export_output.ClassificationOutput(classes=categories_tensor) }
  if mode == tf.estimator.ModeKeys.PREDICT:
    return tf.estimator.EstimatorSpec(
        mode=mode,
        predictions={
            'class': predicted_classes,
            'prob': tf.nn.softmax(logits)
        },
        export_outputs=export_outputs)
  • serving_input_receiver_fn

It needs to be defined on before exporting the trained estimator model, it assumes the input is a raw string and parses your input from there, you can write your own function but I'm unsure whenever you can use external variables. Here is a simple example for a classification algorithm:

def serving_input_receiver_fn():
    feature_spec = { "words": tf.FixedLenFeature(dtype=tf.int64, shape=[4]) }
    return tf.estimator.export.build_parsing_serving_input_receiver_fn(feature_spec)()

  export_dir = classifier.export_savedmodel(export_dir_base=args.job_dir,
                                            serving_input_receiver_fn=serving_input_receiver_fn)

hope it helps.

user_11321500
Related