I am trying to batch predict on the following model:
The input is as follows:
batch_inputs = [<tf.Tensor: shape=(1, 150), dtype=int64, numpy=array([[2, 924, ...]])>, <tf.Tensor: shape=(1, 150), dtype=int64, numpy=array([[2, 130, ...]])>]
If I try to predict while passing multiple inputs like this, it works:
ds = tf.data.Dataset.from_tensor_slices((tf.stack([batch_inputs[0], batch_inputs[0], batch_inputs[0]]))).map(lambda x: {'inputs': x[0], "dec_inputs": x[1]})
predictions = transformer_test.predict(ds)
However the batch size is 1 and thus it takes a longer time that predicting a number of them at the same time. I learnt that since I am using a TF Dataset, the batch size has to be specified in the dataset:
ds = tf.data.Dataset.from_tensor_slices((tf.stack([batch_inputs[0], batch_inputs[0], batch_inputs[0]]))).map(lambda x: {'inputs': x[0], "dec_inputs": x[1]})
predictions = transformer_test.predict(ds.batch(2))
Which returns the following error, since the shape of the input has changed:
ValueError: in user code:
File "/usr/local/lib/python3.7/dist-packages/keras/engine/training.py", line 1801, in predict_function *
return step_function(self, iterator)
File "/usr/local/lib/python3.7/dist-packages/keras/engine/training.py", line 1790, in step_function **
outputs = model.distribute_strategy.run(run_step, args=(data,))
File "/usr/local/lib/python3.7/dist-packages/keras/engine/training.py", line 1783, in run_step **
outputs = model.predict_step(data)
File "/usr/local/lib/python3.7/dist-packages/keras/engine/training.py", line 1751, in predict_step
return self(x, training=False)
File "/usr/local/lib/python3.7/dist-packages/keras/utils/traceback_utils.py", line 67, in error_handler
raise e.with_traceback(filtered_tb) from None
File "/usr/local/lib/python3.7/dist-packages/keras/engine/input_spec.py", line 264, in assert_input_compatibility
raise ValueError(f'Input {input_index} of layer "{layer_name}" is '
ValueError: Exception encountered when calling layer "encoder" (type Functional).
Input 0 of layer "encoder_layer_0" is incompatible with the layer: expected shape=(None, None, 512), found shape=(None, 1, 128, 512)
Call arguments received:
• inputs=['tf.Tensor(shape=(None, 1, 128), dtype=float32)', 'tf.Tensor(shape=(None, 1, 1, 1, 128), dtype=float32)']
• training=False
• mask=None
I created a Google Colab link to reproduce the issue: https://colab.research.google.com/drive/1xYNjLURQcGPIXeWV-xo4_iA09KDJvNfi?usp=sharing