I am trying to understand why my UNet replica model has lower accuracy when I did not downsample at all (Only changing the number of channels throughout the network) compare to when I use maxpooling layer like the original UNet.
Does downsample increase accuracy in general? If so why? Because I thought downsampling would actually reduce the accuracy since you're losing some information of the image.