Batch normalization during testing

user_8384788 0
Viewed 17000

For batch normalization during testing, how does one calculate the mean and variance of each activation input (in each layer and input dimension)? Does one record the means and variances from training, calculate the means and variances of the entire training set, or calculate the means and variances of the entire test set?

Many people say you have to precalculate the means and variances, but if you use the method of calculating the means and variances of the entire test set, wouldn't you need to calculate the means and variances of the entire test set while performing forward propagation (not "pre")?

Thank you so much for all your help!

batch-normalization deep-learning machine-learning neural-network normalization
2 Answers

A record of the empirical mean and variance is taken at training time, such as a running average, which is later used for the test set, instead of calculating the means and variances for each test batch.

user_40728100
Related