Do BatchNormalization and TimeDistributed(BatchNormalization) have the same effect on sequential data (eg video) or not ? If not what's the difference?
1 Answers
In the docs of tf.keras.layers.TimeDistributed, you'll notice,
>> inputs = tf.keras.Input(shape=(10, 128, 128, 3))
>> conv_2d_layer = tf.keras.layers.Conv2D(64, (3, 3))
>> outputs = tf.keras.layers.TimeDistributed(conv_2d_layer)(inputs)
>> outputs.shape
Basically, the layer wrapped in TimeDistributed will be applied to each timestep. Meaning, in the above code example, a Conv2D layer is placed below all 10 timesteps. The same would apply to BatchNormalization.
Instead of TimeDistributed layer, if we apply a BatchNormalization layer directly, the mean and the variance would be computed for all the 10 timesteps as a whole.
Whereas, a BatchNormalization wrapped in a TimeDistributed layer will compute the mean and variance for batches of shape ( 1 , 128 , 128 , 3 ) i.e for each timestep.
Related