Why network with linear layers can't learn anything

user_11085398 0
Viewed 127

I am trying to build a neural network for binary classification, unfortunately it always predicts the value 0, even though one fifth of the training set data is 1. I have no idea why it is so. My dataset looks as this, so there are a couple of categorical variables and a couple of continuous, (target is the one we predict):

enter image description here

Here one can download the data: https://drive.google.com/drive/folders/1PsG2rRdbxyocyqvLSa7zSy_aVDMRJ2Ug?usp=sharing

You can read it with

df = pd.read_csv("train.csv", index_col=0)

Now I am preparing the data for neural network:

x_train=df.drop(labels=['target'], axis=1).values
y_train=df['target'].values
X_train, X_val, y_train, y_val = train_test_split(
x_train,
y_train,
test_size=0.2)

LR = 0.001 
EPOCH = 50
BATCH_SIZE = 64

torch_X_train = torch.tensor(X_train)
torch_y_train = torch.tensor(y_train)

torch_X_val = torch.tensor(X_val)
torch_y_val = torch.tensor(y_val)

train = torch.utils.data.TensorDataset(torch_X_train,torch_y_train)
validate = torch.utils.data.TensorDataset(torch_X_val,torch_y_val)

train_loader = torch.utils.data.DataLoader(train, batch_size = 
BATCH_SIZE, shuffle = True)
val_loader = torch.utils.data.DataLoader(validate, batch_size = 
BATCH_SIZE, shuffle = False)

And define a simple network with just linear layers:

class NN(nn.Module):
def __init__(self, input_size):
    super().__init__()
    self.to_class=nn.Sequential(
    nn.Linear(input_size,512),
    nn.ReLU(),
    nn.Linear(512,256),
    nn.ReLU(),
    nn.Linear(256,32),
    nn.ReLU(),
    nn.Linear(32,2)
    )
def forward(self,inputs):
    pred= self.to_class(inputs)
    return F.softmax(pred, dim=1)

And lastly I train it

net=NN(7)


loss_func = nn.CrossEntropyLoss()
optimizer = optim.Adam(net.parameters(), lr=LR)

train_loss = np.zeros(EPOCH)
val_loss = np.zeros(EPOCH)
acc_train = []
acc_val=[]

for epoch in range(EPOCH):
    correct = 0
    total = 0
    for data in train_loader:
        X, y = data
        optimizer.zero_grad()
        output=net(X.float())
        total += output.size(0)
        pred = output.argmax(dim=1, keepdim=True)
        correct += pred.eq(y.view_as(pred)).sum().item()
        loss = loss_func(output.squeeze(), y) 
        loss.backward()                     
        optimizer.step() 
        train_loss[epoch] += loss
    train_loss[epoch] /= len(train_loader)
    acc_train.append(correct/total*100)
print('epoch %d:\t train_accuracy %.5f\ttrain loss: %.5f'%(epoch,acc_train[epoch], train_loss[epoch]))

But the train loss is not going anywhere and the predictions are always one class! Can someone explain this phenomenon and hint how I can improve it?

enter image description here

pytorch neural-network python-3.x python
1 Answers

Please see the doc

This is the standard training loop

for epoch in range(2):  # loop over the dataset multiple times

    running_loss = 0.0
    for i, data in enumerate(trainloader, 0):
        # get the inputs; data is a list of [inputs, labels]
        inputs, labels = data

        # zero the parameter gradients
        optimizer.zero_grad()

        # forward + backward + optimize
        outputs = net(inputs)
        loss = criterion(outputs, labels)
        loss.backward()
        optimizer.step()

        # print statistics
        running_loss += loss.item()
        if i % 2000 == 1999:    # print every 2000 mini-batches
            print('[%d, %5d] loss: %.3f' %
                  (epoch + 1, i + 1, running_loss / 2000))
            running_loss = 0.0

print('Finished Training')

It seems you are missing optimizer.zero_grad().

Please try to add it and see how it does.

If this doesn't work, please use the same net on a different dataset, for example MNIST, directly downloaded from pytorch. This net should be able to solve MNIST, so this way we can debug if the problem is the net or something related to the data or the labels.

Another thing is balancing the labels: if 80% are 0 and 20% are 1 (or vice versa) you may want to use weighted cross entropy

Another thing you may want is leaky relu instead of relu. This prevents vanishing gradients for some problems.

user_9130980
Related