I've been trying use batch normalization in tensorflow for while with no success. The training loss converges nicely (better than without the BN), but the test loss remains high throughout the training. I'm using batch size of 1 but the problem still happens with bigger batch size. what I'm currently doing is:
inputs = tf.layers.batch_normalization(
inputs=inputs, axis=1 if data_format == 'channels_first' else 3,
momentum== 0.997, epsilon=1e-5, center=True,
scale=True, training=is_training, fused=True)
inputs = tf.nn.relu(inputs)
is_training is tf.placeholder that I assign to True during training and False during testing, and for the training op I do this:
update_ops = tf.get_collection(tf.GraphKeys.UPDATE_OPS)
with tf.control_dependencies(update_ops):
train_op = optimizer.minimize(loss, global_step=batch)
I've tried "tf.contrib.layers.batch_norm" too, and several other implementations of BN that I found online. but nothing works.. I always get the same problem.
I know that the beta and gamma variables are being updated during training. But I also noticed that tf.get_collection(tf.GraphKeys.MOVING_AVERAGE_VARIABLES) is an empty collection, which is weird.
Have anyone seen and solved this problem before? I can't think of any more things to try.
Note: I know the problem is with the BN because without it the test loss converges with the training less as expected.