I have a question regarding the copy and crop arrows in the Unet architecture picture:
I understand the implementation from the contracting and the transfer to the expensive path but I dont get these intra-connections. What should copy and crop mean and what benefit brings this operation?? Is this like a skip connection in ResNet?
It is something similar to the skip connections in ResNet but with some difference.
The architecture difference: In ResNet, the skip connection is adding the activations from the previous block to the next block before calculating activations. These Residual blocks have the effect of skipping the in-between connection by adding output from the previous block.
In U-net the activations or feature maps are merged or concatenated.
Reason for copy or concatenation: Since this is a segmentation problem while downsampling, the spatial information is lost. As you can see it started with 572x572 and reduced to 28x28. In order to regain or map the spatial information back, the feature maps from previous layers are concatenated. If you observe the white block in the upsampling path are the feature maps from the downsampling path.
Reason for crop: Now when merging since the resolution of the two feature maps is not the same they are cropped. Before concatenating (572x572) with (392x392) it needs to be cropped.
The only similarity is utilizing the previously calculated activations but the reasons are different.
The benefit of long skip connections (term used for those long connections) have been studied by Drozdal et al. (https://arxiv.org/pdf/1608.04117.pdf) and have been shown to improve gradient flow and therefore keep the values of the weight matrices stable. It is a kind of hand wavy explanation, but it is the best one we got. However, it does improve results in segmentation tasks.
The crop operation is used in this example, due to the author using valid convolutions, which reduce the output size and therefore the dimension does not fit anymore. In Tensorflow you would use something like tf.image.resize_images to get the same height width dimensions as your feature maps in the expansive part and then use tf.concat to concatenate the feature maps along the channel axis (typically the 4th axis in BHWC layout).
To resize a tensor of shape
(m=batches, h=height, w=width, f=#filters)
use
tf.compat.v1.image.resize_bilinear
or
tf.image.crop_and_resize
In short we are copying the image whose depth is not increased, and croping it to fit it to the image whose depth is already increased.
The main use of it is to retain the features lost when we were increasing depth and going to the image whose width and height were small and the depth was big.
Now we dont know much about where the feature actually belongs to when we are upscaling, so to know that where the feature originally belonged to we are simply copying the earlier stage image which was big and having small depth. Thats it when we were increasing the depth we lost the positional property of the feature and to regain it we are just doing this hack.
Copy and crop don't add the images, it just put the image one on the another, which means the depth would further increase.