We know batch normalization (BN) speeds up training of deep neural networks. But does it help with small neural networks as well? I have been experimenting with a 6-layer convolutional-MLP network and I cannot see any benefit for BN in training this network.
1 Answers
Batch Normalization is generally used for very deep neural networks. The outputs of layers after multiple layers keep fluctuating with every mini-batch and the layer has to keep chasing a moving target.
However, for shallow neural networks, this is not much of a problem, since the fluctuations are within a narrow range and do not pose the problem of a moving target. So for shallow neural networks, you can choose to train without batch normalization too and it will work as expected.
Related