I'm working with GAT model, and I found that many codes in GAT model, they are using xavier_uniform with gain value 1.414.
I wonder they use that specific value.
self.W = nn.Parameter(torch.zeros(size=(in_features, out_features)))
nn.init.xavier_uniform_(self.W.data, gain=1.414)
gain is the the scaling factor in xavier uniform initialization.
root(2) is the recommended gain value for ReLU. So if you're using ReLU activation function, then it's better to use 1.41 as gain value for initialization. Read more here.
gain = nn.init.calculate_gain('relu')
print(gain)
1.4142135623730951
Since the non-linearity will affect the std dev of the activation, you might run into some issues, e.g. vanishing gradients. The gain with respect to the non-linearity should give you “good” statistics of your activations. -- source