I am searching for scientific work on skip-connections.
Everybody is talking about improving gradient flow through the Network, which feels like it makes sense. But I would love to understand and read what that means when it comes down to the math and code and how to in-depth implement this in tensorflow.
Other than the resnet paper I haven't found anything in depth about those magical skip-connections.
I am greatful for all hints, papers and equations.
I am not sure there is anything beyond the fact that it improves Gradient flow which happens because it could easily learn when information from previous layer doesn't have to change much. Apart from that I would say its analogous to RNN's wherein for large time steps it tends to forget the dependencies of the earlier layers. Similarly when you have a CNN network wherein we use pooling layers to reduce the size of the input to grow smaller we do it at the cost of some information so it helps develop a dependency between information we could gain from earlier layers and later layers. Further , what skip connections do is that they help solve an easier optimization problem than without skip connections.
Hope it helps.
