GAN generates exactly the same Images cross a batch only because of seeds distribution, Why?

user_6398487 0
Viewed 963

I have trained a GAN to reproduce CIFAR10 like images. Initially I notice all images cross one batch produced by the generator look always the same, like the picture below: enter image description here

After hours of debugging and comparison to the tutorial which is a great learning source for beginners (https://machinelearningmastery.com/how-to-develop-a-generative-adversarial-network-for-a-cifar-10-small-object-photographs-from-scratch/), I just add only one letter on my original code and the generated images start looking normal (everyone starts looking different from each other cross one batch), like the picture below: enter image description here

The magical one character change on the code is to make the following change:

Changing from:

def generate_latent_points(self, n_samples):
        return np.random.rand(n_samples, self.latent_dim)

to:

def generate_latent_points(self, n_samples):
        return np.random.randn(n_samples, self.latent_dim)

Hope this very subtle detail could help those who spend hours scratching off their head for their GAN training process.

np.random.rand gives uniform distribution over [0, 1)

np.random.randn gives a univariate “normal” (Gaussian) distribution of mean 0 and variance 1

So why the difference in the distribution of the latent seeds for generator would behave so differently?

generative-adversarial-network conv-neural-network neural-network python
2 Answers

There might be few possible reasons that I can think of:

  1. The distribution differences of these two are two folded: 1. rand gives only positive value while randn gives both negative and positive values around 0; 2. rand gives larger value in magnitude than randn for obvious reason. The much larger magnitude might lead to unstable learning because dL/dw (which is in proportional to x) will be larger than that in the case of randn.

  2. The unsuccessful learning of generator puts no challenge on discriminator to distinguish real and fake images, so the discriminator is not learning anything new. (This is my observation on the loss and acc of the discriminator during training process)

user_63984870

I just ran into this problem, found your post, and made the switches. Waiting to see if it improves performance. I can already see more variation in what it is outputting during training, so I'm guessing it will work. Thanks for the tip.

I was printing three random generated images at the end of each epoch, and they were all almost identical, but my generator was slowly switching between patterns that resembled different types of pictures in my sample. I'm guessing that the discriminator was slowly learning the "type" of photos my generator was outputting in a given sequence of batches, so it eventually learned they were fakes, which caused the generator to switch to another type but once again the outputs had almost no variation. The photos were never good by human standards but you could kind of see which images in the real dataset it was trying to imitate and it cycled between those types.

I have a guess as to why your switch improves performance. I believe that the issue is the variance of the distribution. The variance of the Uniform(0,1) distribution is 1/12. The variance of the Standard Normal Distribution is 1. I believe an increase in variance of the input makes it easier for the GAN to learn a high variance image dataset. This is a guess, but it makes sense as the GAN is learning to map from the space of these randomly generated variable to the space of images. If there is a high variability in the output but not the input, the GAN's weights might have to be quite extreme to switch between multiple types of images with only slight variations in input to exploit. It then probably fails with other inputs when going with extreme weights and instead tries to get weights that give it a consistent output no matter the input. As my discriminator saw these were fake it changed it output but remained relatively invariant with the output.

If you wanted to test it, you could choose a Uniform(0,sqrt(12)) which has a variance of 1 and see if it performs as well as the standard normal distribution or better than Uniform(0,1). Variance equal, I wonder if the distribution of the "types" of photos you have matters (obviously a bit abstract). For instance, if the types of photos all appear with similar frequency, will the uniform be better? If the types of photos weren't balanced, would the normal work better? I am guessing having a much larger latent dimension (the input size of your generator) would also accomplish a similar thing but at a greater computational cost.

user_42928500
Related