I have tried to implement both a single Linear Threshold Unit (LTU) and a Perceptron Multiouput Classifier (with two input neurons, three output neurons and one bias neuron) from scratch.
As a point of reference, I am using "Hands-on Machine Learning with Scikit-Learn, Keras & Tensorflow". The perception model is briefly described there but without much detail (except for two equations). No source code gives this additional challenge, but I've been stuck for some time on this (rather) simple implementation. In spite of my best tries, I am still stuck with a very sub-optimal solution. Below I insert code with results and explanation.
The first part is dataset preparation (the iris dataset will all classes). For each class, I use the different output neuron - encoding this into a one-hot matrix (in a not very neat way, but now I am focused on just making things work).
iris = load_iris()
X = iris.data[:, (2, 3)]
y = iris.target.astype(np.int)
y_out1 = np.zeros(150)
y_out1[y == 0] = 1
y_out1 = y_out1.reshape(-1, 1)
y_out2 = np.zeros(150)
y_out2[y == 1] = 1
y_out2 = y_out2.reshape(-1, 1)
y_out3 = np.zeros(150)
y_out3[y == 2] = 1
y_out3 = y_out3.reshape(-1, 1)
y_target = np.concatenate((y_out1, y_out2, y_out3), axis=1)
Next, I am performing proper training using heavside activation function and Hebbian learning. I firstly initialise the matrix with weights and one with bias terms. The rest is quite straightforward. Firstly I compute the outputs of a fully connected layer [heavside(XW + b) then I use those outputs to adjust the weight. According to Hebb's rule, W(i, j)^next = W(i,j) + learning_rate * (y_j, y^{hat}_j) * x_i.
def heavside(z):
z[z < 0] = 0
z[z > 0] = 1
return z
W = np.random.default_rng().uniform(low = -1, high = 1, size=(2, 3)) # One row per input neuron and one column per artifical neuron
b = np.ones((1, 3)) # One bias term per artifical neuron.
epoch = 2
learning_rate = 0.11
for epoch in range(epochs):
z = X.dot(W) + b
predictions = heavside(z)
for k in range(len(X)):
for i in range(W.shape[0]):
for j in range(W.shape[1]):
W[i][j] = W[i][j] + learning_rate * (y_target[k][j] - predictions[k][j]) * X[k, i]
According to the materials provided in the book, if the training instances are linearly separable (which they generally are), the algorithm should converge to a solution. However, provided predictions are far from any sensible output:
array([[1., 1., 0.],
[1., 1., 0.],
[1., 1., 0.],
[1., 1., 0.],
[1., 1., 0.],
[1., 1., 0.],
[1., 1., 0.],
[1., 1., 0.],
[1., 1., 0.],
[1., 1., 0.],
[1., 1., 0.],
[1., 1., 0.],
[1., 1., 0.],
[1., 1., 0.],
[1., 1., 0.],
[1., 1., 0.],
[1., 1., 0.],
[1., 1., 0.],
[1., 1., 0.],
[1., 1., 0.],
[1., 1., 0.],
[1., 1., 0.],
[1., 1., 0.],
[1., 1., 0.],
[1., 1., 0.],
[1., 1., 0.],
[1., 1., 0.],
[1., 1., 0.],
[1., 1., 0.],
[1., 1., 0.],
[1., 1., 0.],
[1., 1., 0.],
[1., 1., 0.],
[1., 1., 0.],
[1., 1., 0.],
[1., 1., 0.],
[1., 1., 0.],
[1., 1., 0.],
[1., 1., 0.],
[1., 1., 0.],
[1., 1., 0.],
[1., 1., 0.],
[1., 1., 0.],
[1., 1., 0.],
[1., 1., 0.],
[1., 1., 0.],
[1., 1., 0.],
[1., 1., 0.],
[1., 1., 0.],
[1., 1., 0.],
[0., 1., 0.],
[0., 1., 0.],
[0., 1., 0.],
[0., 1., 0.],
[0., 1., 0.],
[0., 1., 0.],
[0., 1., 0.],
[0., 1., 0.],
[0., 1., 0.],
[0., 1., 0.],
[0., 1., 0.],
[0., 1., 0.],
[0., 1., 0.],
[0., 1., 0.],
[0., 1., 0.],
[0., 1., 0.],
[0., 1., 0.],
[0., 1., 0.],
[0., 1., 0.],
[0., 1., 0.],
[0., 1., 0.],
[0., 1., 0.],
[0., 1., 0.],
[0., 1., 0.],
[0., 1., 0.],
[0., 1., 0.],
[0., 1., 0.],
[0., 1., 0.],
[0., 1., 0.],
[0., 1., 0.],
[0., 1., 0.],
[0., 1., 0.],
[0., 1., 0.],
[0., 1., 0.],
[0., 1., 0.],
[0., 1., 0.],
[0., 1., 0.],
[0., 1., 0.],
[0., 1., 0.],
[0., 1., 0.],
[0., 1., 0.],
[0., 1., 0.],
[0., 1., 0.],
[0., 1., 0.],
[0., 1., 0.],
[0., 1., 0.],
[0., 1., 0.],
[0., 1., 0.],
[0., 1., 0.],
[0., 1., 0.],
[0., 1., 0.],
[0., 1., 0.],
[0., 1., 0.],
[0., 1., 0.],
[0., 1., 0.],
[0., 1., 0.],
[0., 1., 0.],
[0., 1., 0.],
[0., 1., 0.],
[0., 1., 0.],
[0., 1., 0.],
[0., 1., 0.],
[0., 1., 0.],
[0., 1., 0.],
[0., 1., 0.],
[0., 1., 0.],
[0., 1., 0.],
[0., 1., 0.],
[0., 1., 0.],
[0., 1., 0.],
[0., 1., 0.],
[0., 1., 0.],
[0., 1., 0.],
[0., 1., 0.],
[0., 1., 0.],
[0., 1., 0.],
[0., 1., 0.],
[0., 1., 0.],
[0., 1., 0.],
[0., 1., 0.],
[0., 1., 0.],
[0., 1., 0.],
[0., 1., 0.],
[0., 1., 0.],
[0., 1., 0.],
[0., 1., 0.],
[0., 1., 0.],
[0., 1., 0.],
[0., 1., 0.],
[0., 1., 0.],
[0., 1., 0.],
[0., 1., 0.],
[0., 1., 0.],
[0., 1., 0.],
[0., 1., 0.],
[0., 1., 0.],
[0., 1., 0.],
[0., 1., 0.],
[0., 1., 0.],
[0., 1., 0.]])