I'm building an image translation network that takes a singe image and outputs multiple translated images. The training data would therefore be (x, Y), x (3, H, W) is the input the image, Y (3*N, H, W) is the concatenated tensor of N target images ys.
I'm wondering how to normalise the data. Should I normalise x and each y individually, or should I use the the overall mean/standard deviation to normalise them collectively?