how can I do a back propagation for a simple neural net with sigmoid activation?

Viewed 107

I am a beginner in deep learning. I am currently struggled with the back propagation algorithms. I found this piece of code online of the back propagation of a simple neural net with sigmoid activation function.

#Step 1 Collect Data
x = np.array([[0,0,1], [0,1,1], [1,0,1], [1,1,1]])
y = np.array([[0], [1], [1], [0]])

#Step 2 build model
num_epochs = 60000

#initialize weights
syn0 = 2np.random.random((3,4)) - 1 
syn1 = 2np.random.random((4,1)) - 1

def nonlin(x,deriv=False):
    if(deriv==True): return x*(1-x)
        return 1/(1+np.exp(-x))
    for j in xrange(num_epochs): #feed forward through layers 0,1, and 2
        k0 = x
        k1 = nonlin(np.dot(k0, syn0))
        k2 = nonlin(np.dot(k1, syn1))

        #how much did we miss the target value?
        k2_error = y - k2

        if (j% 10000) == 0:
            print "Error:" + str(np.mean(np.abs(k2_error)))

        #in what direction is the target value?
        k2_delta = k2_error*nonlin(k2, deriv=True)

        #how much did each k1 value contribute to k2 error
        k1_error = k2_delta.dot(syn1.T)
        k1_delta= k1_error * nonlin(k1,deriv=True)
        syn1 += k1.T.dot(k2_delta)
        syn0 += k0.T.dot(k1_delta)

I do not get this line of code: k2_delta = k2_error*nonlin(k2, deriv=True). When calculating the local gradient why it uses k2_error multiply the derivative of k2. Should we use a different thing instead of k2_error because the cost function in this algorithm is absolute value, so should I use a vector of [-1,1,1,-1] as the local gradient of the cost function? I assume here it uses analytics gradient.

2 Answers
Related