Keras Matrix Multiplication to get Predicted Values

user_3737798 0
Viewed 1188

I'm looking to take the output of a Keras model to manually calculate the predicted values through matrix multiplication. I would like to do this to help understand how Keras is working under the hood. I'll use the simple XOR problem. Here is my code:

import numpy as np
import keras
from keras.models import Sequential
from keras.layers.core import Dense
from keras.callbacks import LambdaCallback

class LossHistory(keras.callbacks.Callback):
    def on_train_begin(self, logs={}):
        self.losses = []

    def on_batch_end(self, batch, logs={}):
        self.losses.append(logs.get('loss'))


history = LossHistory()

# the four different states of the XOR gate
training_data = np.array([[0,0],[0,1],[1,0],[1,1]], "float32")

# the four expected results in the same order
target_data = np.array([[0],[1],[1],[0]], "float32")

model = Sequential()
model.add(Dense(4, input_dim=2, activation='relu'))
model.add(Dense(1, activation='sigmoid'))

print_weights = LambdaCallback(on_epoch_end=lambda batch, logs: print(model.layers[0].get_weights()))

model.compile(loss='mean_squared_error',
              optimizer='adam',
              metrics=['binary_accuracy'])

history2 = model.fit(training_data, target_data, epochs=50, verbose=2, callbacks=[print_weights, history])

print(model.predict(training_data).round())


W1 = model.get_weights()[0]
X1 = np.matrix([[0,0],[1,1]], "float32")
wx = np.dot(X1,W1)
b = model.get_weights()[1]
wx = np.reshape(wx,(4,2))
b = np.reshape(b, (4,1))
z = wx + b
from numpy import array, exp
a1 = 1 / (1 + exp(-z))
print('g =\n', a1)

W2 = model.get_weights()[2]
b2 = model.get_weights()[3]
W2 = np.reshape(W2,(1,4))
a1 = np.reshape(a1, (4,1))
wa = np.dot(W2,a1)
z2 = wa + b2
a2 = 1 / (1 + exp(-z2))
print('g =\n', a2)

From what I understand, get_weights()[0] and get_weights()[1] are the weights and biases for the first layer, respectively, and get_weights()[2] and get_weights()[3] are the weights and biases for the second layer. I believe the issue I'm have is figuring out what x1 and x2 are as they relate to the equation z = Wx + b. The weights are retrieved from the last epoch and are usually weights that achieve 100% accuracy. The output I'm expecting is [0,1,1,0] for the y-hat predictions based on the manual calculation of z = Wx + b and then taking the sigmoid of z.

keras xor neural-network matrix python
1 Answers

You are very close!

First off, 50 epochs using a training set with only 4 events was not enough to replicate a consisting correct output (0,1,1,0), so I bumped the number of epochs to 1000. Below is the code I used with the decimal and rounded outputs:

import numpy as np
from keras.models import Sequential
from keras.layers.core import Dense

# Set seed for reproducibility
np.random.seed(1)

# the four different states of the XOR gate
training_data = np.array([[0,0],[0,1],[1,0],[1,1]], "float32")
# the four expected results in the same order
target_data = np.array([[0],[1],[1],[0]], "float32")

model = Sequential()
model.add(Dense(4, input_dim=2, activation='relu'))
model.add(Dense(1, activation='sigmoid'))
model.compile(loss='mean_squared_error',optimizer='adam',metrics=['binary_accuracy'])

history = model.fit(training_data, target_data, epochs=1000, verbose=1)

# decimal output
print('decimal output:\n'+str(model.predict(training_data)))
# rounded output
print('rounded output:\n'+str(model.predict(training_data).round()))
# ouputs:
decimal output:
[[ 0.25588933]
 [ 0.82657152]
 [ 0.83840138]
 [ 0.16465074]]
rounded output:
[[ 0.]
 [ 1.]
 [ 1.]
 [ 0.]]

The model is giving the correct rounded output, good! The decimal output is nice to use for comparing the manual approach.

For the manual approach, X1 is the input to the model, either [0,0], [0,1], [1,0] or [1,1]. X2 is the output of the first layer, and input to the last layer. The weight and biases are exactly as you said ("get_weights()[0] and get_weights()[1] are the weights and biases for the first layer, respectively, and get_weights()[2] and get_weights()[3] are the weights and biases for the second layer"). But it may seem like you forgot the relu activation function in the first layer? Lets look at the solution code:

# Parameters layer 1
W1 = model.get_weights()[0]
b1 = model.get_weights()[1]

# Parameters layer 2
W2 = model.get_weights()[2]
b2 = model.get_weights()[3]

# Input
X1 = np.array([[0,0],[0,1],[1,0],[1,1]], "float32")
# Use the following X1 for single input instead of all at once
#X1 = np.array([[0,0]])

# First layer calculation
L1 = np.dot(X1,W1)+b1
# Relu activation function
X2 = np.maximum(L1,0)
# Second layer calculation
L2 = np.dot(X2,W2)+b2
# Sigmoid
output = 1/(1+np.exp(-L2))

# decimal output
print('decimal output:\n'+str(output))
# rounded output
print('rounded output:\n'+str(output.round()))
# ouputs:
decimal output:
[[ 0.25588933]
 [ 0.82657152]
 [ 0.83840144]
 [ 0.16465074]]
rounded output:
[[ 0.]
 [ 1.]
 [ 1.]
 [ 0.]]

You can use all 4 input simultaneously as above, or just a single input as the commented out #X1 suggest. Note that the decimal "model.predict" output, and the manual approach gives the exact same output (with a small deviation at the third value, probably due to some keras/numpy rounding deviation?)

user_104554930
Related