How does backpropagation changes pixels (input data) in neural style transfer?

user_11620429 0
Viewed 191

I am quite new to machine learning and I want to understand Neural Style Transfer better, but I feel I'm missing something. As far as I know

  1. Backpropagation updates weights, not inputs. The generated image (G) is an input
  2. Neural Style Transfer approach involves transfer learning, meaning it uses a pretrained network (VGG) in order to calculate the cost function. This network isn't used to learn something else, but to give insights about the closeness between G and Content Image (C) and between G and Style Image (S)

My question is: how is the image updated? I may be very wrong, but updating the network seems to ruin the insights that helped defining the cost function in the first place (cost function uses activations of some layers in order to determine if G has a similar content with C and a style close to S). However, let's say we alter the network. But that means changing the weights, not the input. How do we get to adjust the pixels in relation with the cost function?

I've already watched https://www.youtube.com/watch?v=xY-DMAJpIP4&list=PLkDaE6sCZn6Gl29AoE31iwdVwSG-KnDzF&index=39 and the other videos from Andrew Ng about the subject, but I still don't get it. I mention I am referring to the first approach of Neural Style Transfer (Leon A. Gatys et al.)

backpropagation conv-neural-network neural-network
1 Answers

I also had the same doubt while going through the programming tutorial on neural style transfer in the cnn course of deeplearning.ai. Things became clearer when I went through neural style transfer tutorial on tensorflow.

In the tutorial, generated image G is declared as tf.variable. So imagine the pixels of G as weights. Hence on gradient descent pixel gets updated just as weights gets updated.

user_132920490
Related