Difference between WGAN and WGAN-GP (Gradient Penalty)

user_16277253 0
Viewed 451

I just find that in the code here:

https://github.com/NUS-Tim/Pytorch-WGAN/tree/master/models

The "generator" loss, G, between WGAN and WGAN-GP is different, for WGAN:

g_loss = self.D(fake_images)
g_loss = g_loss.mean().mean(0).view(1)
g_loss.backward(one) # !!!
g_cost = -g_loss

But for WGAN-GP:

g_loss = self.D(fake_images)
g_loss = g_loss.mean()
g_loss.backward(mone) # !!!
g_cost = -g_loss

Why one is one=1 and another is mone=-1?

backpropagation generative-adversarial-network pytorch deep-learning neural-network
2 Answers

You might have misread the source code, the first sample you gave is not averaging the resut of D to compute its loss but instead uses the binary cross-entropy.

To be more precise:

  • The first method ("GAN") uses the BCE loss to compute the loss terms for D and G. The standard GAN optimization objective for D is to minimize E_x[log(D(x))] + E_z[log(1-D(G(z)))].
    Source code:

    outputs = self.D(images)
d_loss_real = self.loss(outputs.flatten(), real_labels) # <- bce loss
real_score = outputs

# Compute BCELoss using fake images
fake_images = self.G(z)
outputs = self.D(fake_images)
d_loss_fake = self.loss(outputs.flatten(), fake_labels) # <- bce loss
fake_score = outputs

# Optimizie discriminator
d_loss = d_loss_real + d_loss_fake
self.D.zero_grad()
d_loss.backward()
self.d_optimizer.step()

    For d_loss_real you optimize towards 1s (output is considered real), while d_loss_fake optimizes towards 0s (output is considered fake).

  • While the second ("WCGAN") uses the Wasserstein loss (ref) whereby we maximise for D the loss: E_x[D(x)] - E_z[D(G(z))].
    Source code:

    # Train discriminator
# WGAN - Training discriminator more iterations than generator
# Train with real images
d_loss_real = self.D(images)
d_loss_real = d_loss_real.mean()
d_loss_real.backward(mone)

# Train with fake images
z = self.get_torch_variable(torch.randn(self.batch_size, 100, 1, 1))

fake_images = self.G(z)
d_loss_fake = self.D(fake_images)
d_loss_fake = d_loss_fake.mean()
d_loss_fake.backward(one)

# [...]

Wasserstein_D = d_loss_real - d_loss_fake

    By doing d_loss_real.backward(mone) you backpropage with a gradient of opposite sign, i.e. its's a gradient ascend, and you end up maximizing d_loss_real.

user_63313690

In order to Update D network: lossD = Expectation of D(fake data) - Expectation of D(real data) + gradient penalty lossD ↓，D(real data) ↑

so you need to add minus one to the gradient process

user_178522200
Related