I am a beginner in deep learning. I am currently struggled with the back propagation algorithms. I found this piece of code online of the back propagation of a simple neural net with sigmoid activation function.
#Step 1 Collect Data
x = np.array([[0,0,1], [0,1,1], [1,0,1], [1,1,1]])
y = np.array([[0], [1], [1], [0]])
#Step 2 build model
num_epochs = 60000
#initialize weights
syn0 = 2np.random.random((3,4)) - 1
syn1 = 2np.random.random((4,1)) - 1
def nonlin(x,deriv=False):
if(deriv==True): return x*(1-x)
return 1/(1+np.exp(-x))
for j in xrange(num_epochs): #feed forward through layers 0,1, and 2
k0 = x
k1 = nonlin(np.dot(k0, syn0))
k2 = nonlin(np.dot(k1, syn1))
#how much did we miss the target value?
k2_error = y - k2
if (j% 10000) == 0:
print "Error:" + str(np.mean(np.abs(k2_error)))
#in what direction is the target value?
k2_delta = k2_error*nonlin(k2, deriv=True)
#how much did each k1 value contribute to k2 error
k1_error = k2_delta.dot(syn1.T)
k1_delta= k1_error * nonlin(k1,deriv=True)
syn1 += k1.T.dot(k2_delta)
syn0 += k0.T.dot(k1_delta)
I do not get this line of code: k2_delta = k2_error*nonlin(k2, deriv=True). When calculating the local gradient why it uses k2_error multiply the derivative of k2. Should we use a different thing instead of k2_error because the cost function in this algorithm is absolute value, so should I use a vector of [-1,1,1,-1] as the local gradient of the cost function? I assume here it uses analytics gradient.