I built and trained a model using TensorFlow 2.2.
When loading the model and using model.predict, I get deterministic results back. This is great.
When using TensorFlow Serving, I always get different predictions back.
My model does have a couple of Dropout layers. However, I saved the trained weights and loaded them into another model that doesn't have the Dropout layers. Save non-deterministic results.
Here is the model architecture:
base_model = NASNetLarge(
include_top=False,
weights=None,
classes=453,
input_tensor=Input(shape=(331, 331, 3), name="image"))
x = base_model.output
x = Dense(128, activation='relu')(x)
x = GlobalAveragePooling2D()(x)
x = Dropout(0.2)(x)
predictions = Dense(2014, activation='softmax')(x)
source_model = Model(base_model.input, predictions)
input2 = Input(shape=(1, ), name="orientation")
x2 = Dense(1, activation="relu")(input2)
concatenated = concatenate([source_model.layers[-2].output, x2])
x3 = Dense(64, activation="relu")(concatenated)
x3 = Dropout(0.2)(x3)
predictions = Dense(453, activation="softmax")(x3)
model = Model(inputs=[base_model.input, input2], outputs=predictions)