I am dealing with an image-to-image regression problem. I want to use CNN (with a U-shape) to learn the mapping between X and Y at resolution 32x32. Once trained, given inputs X at resolution 320x320, one can obtain the prediction for Y at resolution 320x320.
The objective is to enhance the spatial resolution of Y. Note that this can not be realized by image super-resolution as no high-resolution (320x320) Y images available.
My question is that despite CNN performs well at low resolution (32x32), its predictions for high-resolution Y have very similar spatial patterns with X. But it shouldn't be like that.. Even if I crop the high-resolution X to give 32x32 patches as inputs to CNN, the results are still similar.
Any suggestions would be greatly appreciated!