If a ConvLSTM layer has 4 filters, then the output shape should be (timestamp, height, width, 4). However, at every timestamp, the convolutional operation makes the hidden state matrix a 4-channel output. I am wondering how a 4-channel output is multiplied by the original weight matrix to get the hidden state at the next timestamp since the original weight matrix shape doesn't match the output hidden state shape.
Here is an example using TensorFlow:
import os
os.environ['TF_CPP_MIN_LOG_LEVEL'] = '2'
import tensorflow as tf
from tensorflow import keras
from tensorflow.keras import layers
import numpy as np
np.set_printoptions(suppress=True,threshold=np.inf,formatter={'float_kind':'{:16.7f}'.format})
tf.keras.backend.set_floatx('float64')
in0Con52256 = tf.keras.layers.Input(shape=([2, 2, 2, 1]))
Con52256 = keras.layers.ConvLSTM2D(4, (2, 1),recurrent_activation='linear', activation='linear', return_sequences=True, name = 'Con52256', )(in0Con52256)
model = tf.keras.models.Model(inputs=[in0Con52256], outputs=Con52256)
w = model.get_layer('Con52256').get_weights()
w[0] = np.array([[[[0.824, 0.2379, 0.7184, 0.7281, 0.3059, 0.5749, 0.6929, 0.702, 0.8117, 0.1209, 0.1804, 0.9257, 0.8759, 0.1887, 0.5883, 0.0049]]], [[[0.856, 0.4612, 0.9902, 0.7742, 0.4632, 0.2713, 0.3251, 0.3908, 0.3784, 0.8545, 0.6903, 0.654, 0.57, 0.2927, 0.3499, 0.7751]]]])
w[1] = np.array([[[[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1], [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1], [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1], [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]]], [[[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1], [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1], [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1], [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]]]])
w[2] = np.array([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0])
model.get_layer('Con52256').set_weights(w)
in0Con52256 = tf.constant([[[[[0.7109], [0.8732]], [[0.7737], [0.6371]]], [[[0.4515], [0.2206]], [[0.9161], [0.407]]]]])
print (np.array2string(model.predict([in0Con52256],steps=1), separator=', '))
The output is:
[[[[[ 1.1547137, 0.1416927, 0.5826253,
0.7840216],
[ 1.3552120, 0.1145092, 0.5535885,
0.6888811]]],
[[[ 58.7095432, 37.4722277, 51.7272749,
60.6180669],
[ 42.1665651, 28.9434071, 36.3540738,
43.4047547]]]]]