Why is my ReLU activation giving wrong outputs?

Admin 1
Viewed 180

The ReLU function, in this case I used the leaky ReLU is giving me wrong output. But using the sigmoid function it gives me an acceptable output

Here is the code that I have: 

import numpy as np

def relu(x):
    return np.maximum(0.01 * x, x)


def relu_derivative(x):
    x[x>0] = 1
    x[x<0] = 0.01
    return x

training_inputs = np.array([[1, 0], 
                            [1, 1], 
                            [0, 0], 
                            [0, 1]])
training_outputs = np.array([[1, 0, 0, 1]]).T

weights = 2 * np.random.random((2, 1)) - 1

print('Weights before training: ')
print(weights)

for epochs in range(10000):
    outputs = relu(np.dot(training_inputs, weights))

    error = training_outputs - outputs

    adjustment = error * relu_derivative(outputs)

    weights += np.dot(training_inputs.T, adjustment)

print('Neuron Weights after training: ')
print(weights)

print('Outputs after training: ')
print(outputs)

Epochs = 10000

Outputs after training using the ReLU function = [0.01],[0.01],[0.01],[0.01],[0.01]

Outputs after training using the sigmoid function = [0.99],[0.01],[0.006],[0.99],[0.98]

The output given by the sigmoid function is better than ReLU's and I tested with epochs up to 100000 and the result for ReLU function is still the same. Is there something wrong with my function or code?

machine-learning python
1 Answers

First a small error in your relu_derivative function. You shouldn't modify x value but rather create a new array:

def relu_derivative(x):
    y = np.zeros_like(x)
    y[x>0] = 1
    y[x<0] = 0.01
    return y

But it does not answers to your question as it still does not learn properly how to solve the XOR. What I think is that 1 hidden unit is not enough for this problem with relu.

I rewrote the same experiment with PyTorch, here is the code:

import torch


class Model(torch.nn.Module):
    def __init__(self, *args, **kwargs):
        super().__init__()
        self.hidden = torch.nn.Linear(2, kwargs['h'])
        self.relu = torch.nn.LeakyReLU(0.1)
        self.out = torch.nn.Linear(kwargs['h'], 1)

        with torch.no_grad():
            self.hidden.bias.zero_()
            self.out.bias.zero_()

    def forward(self, x):
        z = self.hidden(x)
        z = self.relu(z)
        z = self.out(z)
        return z


if __name__ == '__main__':
    training_inputs = torch.Tensor([[1., 0.],
                                    [1., 1.],
                                    [0., 0.],
                                    [0., 1.]])
    training_outputs = torch.Tensor([1., 0., 0., 1.]).reshape(4, 1)

    model = Model(h=2)
    learning_rate = 0.01

    criterion = torch.nn.MSELoss()
    optimizer = torch.optim.Adam(model.parameters(), lr=learning_rate)
    print(*[i for i in model.parameters()], sep='\n')
    for _ in range(1000):
        pred = model(training_inputs)
        loss = criterion(pred, training_outputs)

        optimizer.zero_grad()
        loss.backward()
        optimizer.step()

    print(pred, loss)
    print(*[i for i in model.parameters()], sep='\n')

And indeed with only 1 hidden unit, it seems that you cannot solve the XOR, but with 2 hidden units, sometimes (depending on initialization) it works.

user_81180870
Related