Somehow for my implementation, I have to define the weights first and can not use high level functions in tensorflow like tf.layers.batch_normalization or tf.layers.dense. So to do batch normalization I need to use tf.nn.batch_normalization. I know that for calculating mean and variance of each minibatch I can use tf.nn.moments, but what about moving mean and variance? Does anyone have experience with doing that or knows an example of implementation? I see that people talk about using tf.nn.batch_normalization could be tricky so I want to know the complexity of doing that. In other words, what makes it tricky and what points I should be careful about during my implementation? Is there any other point besides moving average and variance that I should be aware of?
1 Answers
You have to be wary of the terms running_mean and running_variance. In mathematics and in traditional cocmputer science, they are referred to as methods which compute these values without having seen the complete data. They are also known as online versions of mean and variance. Not that they are able to accurately determine the mean and variance beforehand. They simply keep on updaing the value of some variables mean and variance, as more data comes in. If your data size is finite, then once having seen the complete data, their values would match the values one would compute, if the complete data is made available.
The case of batch normalization is different. You should not think of running mean and running variance in the same way as in the above paragraph.
Training Time
During training, the mean and variance are computed for a batch. They is not running mean or running variance. So, you can safely use tf.nn.moments to do that.
Testing Time
During testing time, you use what should be called as population_estimated_mean and population_estimated_variance. These quantities are computed during training but are not directly used. They are computed for later use during testing time.
Now one pitfall is that, some people might want to use the Knuth Formula for computing these quantities. This is not advisable. Why? : Because, training is done over several epochs. So, the same dataset is seen for as many times as the number of epochs. Since data augmentation is also usually random, computing the standard running mean and running variance, could be dangerous. Instead what is usually used is an exponentially decaying estimate.
You can achieve this, by using tf.train.ExponentialMovingAverage over batch_mean and batch_variance. Here you specify that how much relevance will be given to past samples vis-a-vis present samples. Be sure that the variables you use for computing this should be non-trainable by setting trainable=False.
During test time, you will use these variables as mean and variance.
For more details on implementation you can take a look at this link.
Related