Does Back Propagation and Gradient Descent use the same logic?

Viewed 328

Back Propagation is used in CNN to update the randomly allotted weights, biases and filters. For updation of values, we find Gradient using chain rule from end to start and use the formula,

New Value = old value - (learning Rate * gradient)

Gradient Descent is an optimiser, which is used to optimize the loss functions. Here also gradient is calculated and the formula is

New value = old value - (learning Rate * gradient)

Correct me if I am wrong in the above explanation given.

My Doubts are:

  • Does both Back propagation and Gradient Descent use the same logic?
  • Is there any relation between Back Propagation and Gradient Descent ?
2 Answers

So you are getting confused, back-propagation is just the name for reversing the order that you move by, usually in forward-propagation you move forward through each layer, but in back-propagation you move backwards.

You can think of propagation as the engine of a neural network, in forward propagation it moves forward, and in back-propagation it moves backwards. The engine doesn't do any of the calculations, it is only used to define which layer you are on, and which layer is next.

Within back-propagation you have cost-functions. One of the types of cost functions you can use is called the Gradient Descent algorithm. Another one could be the Stochastic Gradient Descent algorithm. These algorithms are used to find the best values to adjust the weights by.

I hope this resolves your question, if you need any more information just leave a comment. If you want more information regarding the details of Gradient Descent or Stochastic Gradient Descent, I recommend you check some of my previous answers here and here.

Bye and have a good day!

Backpropagation is gradient descent for neural networks. Gradient descent is the more general first-order iterative optimization algorithm to approximate differentiable functions. Using it to train multi-layer neural networks (where each layer represents a differentiable function), we can derive a more specialized algorithm for this type of model (dealing with the fact that we have hidden units). The resulting algorithm is called Backpropagation.

There's a second distinction that can be made between SGD and GD in the specific context of how much data you show per training iteration (see James Barnett's answer), though I would argue that the term "mini-batch learning" is more precise if one wants to contrast it to stochastic gradient descent.

Related