I am trying to create a Scaled Dot Attention model using keras utility layer collection found at https://github.com/zimmerrol/keras-utility-layer-collection. I am using the default model (code below) provided in the examples notebook in the github repo. My problem is I get the following error :
ValueError: Input 0 is incompatible with layer model_1: expected shape=(None, 500, 3), found shape=(500, 3)
The code is below
# input: time series with 500 steps
# each step has a 3dim valuethe output sequences
# of a LSTM, RNN, etc.
net_input = Input(shape=(500, 3))
net = TimeDistributed(Dense(3))(net_input)
# queries
net_q = TimeDistributed(Dense(3))(net_input)
# values
net_v = TimeDistributed(Dense(3))(net_input)
# keys
net_k = TimeDistributed(Dense(3))(net_input)
# add one ScaledDotProductAttention layer
net = attention.ScaledDotProductAttention(name="attention", return_attention=False)([net_q, net_v, net_k])
net_output = TimeDistributed(Dense(2))(net)
model = Model(inputs=net_input, outputs=net_output)
model.summary()
model.compile(loss="categorical_crossentropy", optimizer="adam", metrics='accuracy')
# dummy data
# x = np.random.rand(64, 500, 3)
# y = np.random.rand(64, 500, 2)
# X, y = make_multilabel_classification(n_samples=500, n_features=3, n_classes=3, n_labels=2, random_state=1)
X, y = make_classification(n_samples=500, n_features=3, n_redundant=0, n_repeated=0, n_informative=3,
n_classes=3, random_state=1)
print(X.shape, y.shape)
model.fit(X, y, batch_size=500, epochs=1)
EDIT : The summary of the model is also added below
Model: "model"
__________________________________________________________________________________________________
Layer (type) Output Shape Param # Connected to
==================================================================================================
input_2 (InputLayer) [(None, 500, 3)] 0
__________________________________________________________________________________________________
time_distributed_5 (TimeDistrib (None, 500, 3) 12 input_2[0][0]
__________________________________________________________________________________________________
time_distributed_6 (TimeDistrib (None, 500, 3) 12 input_2[0][0]
__________________________________________________________________________________________________
time_distributed_7 (TimeDistrib (None, 500, 3) 12 input_2[0][0]
__________________________________________________________________________________________________
attention (ScaledDotProductAtte (None, 500, 3) 0 time_distributed_5[0][0]
time_distributed_6[0][0]
time_distributed_7[0][0]
__________________________________________________________________________________________________
time_distributed_8 (TimeDistrib (None, 500, 2) 8 attention[0][0]
==================================================================================================
Total params: 44
Trainable params: 44
Non-trainable params: 0
__________________________________________________________________________________________________
Any help would be appreciated. Thanks!