Can I use Layer Normalization with CNN?

user_7329645 0
Viewed 8962

I see the Layer Normalization is the modern normalization method than Batch Normalization, and it is very simple to coding in Tensorflow. But I think the layer normalization is designed for RNN, and the batch normalization for CNN. Can I use the layer normalization with CNN that process image classification task? What are the criteria for choosing batch normalization or layer?

data-processing batch-normalization tensorflow deep-learning normalization
3 Answers

In more recent work [1], it was found that you can use LayerNorm in CNNs without degrading accuracy, though it depends on the model architecture. Liu et al. [1] found while developing ConvNeXt that "Directly substituting LN for BN in the original ResNet will result in suboptimal performance" but they observed that their ConvNeXt "model does not have any difficulties training with LN; in fact, the performance is slightly better".

It would be great if there were a better explanation as to why...

  1. Liu et al. A ConvNet for the 2020s. https://arxiv.org/pdf/2201.03545.pdf
user_80385610
Related