clip_grad_norm_ performs gradient clipping, in order to mitigate the problem of exploding gradients.
BatchNorm2d applies Batch Normalization (for the same reason - mitigate the problem of exploding gradients)
I know that BatchNorm2d has 2 parameters to learn (mean and standard deviation).
When we should choose clip_grad_norm_ and when we should prefer BatchNorm2d?
As far as I know, BatchNorm2d is used to mitigate too different scale and shift of the input data on each layer in the network. However, BatchNorm2d doesn't do anything with the gradients directly. Although I heard that BatchNorm2d sometimes becomes a cause of gradient explosion. Not sure if it can mitigate gradient explosion instead.
So, it seems that clip_grad_norm_ and BatchNorm2d are two very different things and there is no reason to choose one for solving the same problem. If there is a potential gradient explosion problem then try gradient clipping. If there is an overfit because of variety in layer input distributions then try BatchNorm2d.
