How are the non-trainable parameters in the summary coming from preprocessing.Normalization in tensorflow.keras.layers.experimental calculated?

user_5102942 0
Viewed 208

I have been using preprocessing.Normalization in tensorflow.keras.layers.experimental to normalize data. Here, when I use one input ([None,1]), I get 3 non-trainable parameters in the summary. When I use nine inputs ([None,9]), I get 19 non-trainable parameters. These examples, summary, code and complete data can be found here. How the "9" and the "19" are calculated?

I have seen a similar question for BatchNormalization but it is likely that it is not the same case for the above-mentioned normalization. I would like to know in more detail what they mean and how are they calculated, is it related somehow to the mean, standard deviation and bias? What is the theory behind it? Can someone point to a paper?

Thanks.

keras tensorflow deep-learning python
1 Answers

I do not know the details of the Keras implementation, but the normalization should be something like (from https://keras.io/api/layers/normalization_layers/layer_normalization/):

x_i_normalized = (x_i - mean_i) / sqrt(var_i + epsilon)

So these untrainable parameters are probably the mean and standard deviation of the individual inputs, so 2 times the number of inputs 2 * n, and then one additional parameter to avoid dividing by a zero, i.e. 2 * n + 1.

user_112417750
Related