Initial bias values for a neural network

user_8149948 0
Viewed 11143

I am currently building a CNN in tensorflow and I am initialising my weight matrix using a He normal weight initialisation. However, I am unsure how I should initialise my bias values. I am using ReLU as my activation function between each convolutional layer. Is there a standard method to initialising bias values?

# Define approximate xavier weight initialization (with RelU correction described by He)
def xavier_over_two(shape):
    std = np.sqrt(shape[0] * shape[1] * shape[2])
    return tf.random_normal(shape, stddev=std)

def bias_init(shape):
    return #???
bias-neuron tensorflow machine-learning
2 Answers

Be aware of the specific case of the last layer's bias. As Andrej Karpathy explains in his Recipe for Training Neural Networks:

init well. Initialize the final layer weights correctly. E.g. if you are regressing some values that have a mean of 50 then initialize the final bias to 50. If you have an imbalanced dataset of a ratio 1:10 of positives:negatives, set the bias on your logits such that your network predicts probability of 0.1 at initialization. Setting these correctly will speed up convergence and eliminate “hockey stick” loss curves where in the first few iteration your network is basically just learning the bias.

user_87290730
Related