My Loss Function doesn't get smaller values during training

Question

I am trying to predict the center of my palm

The structure of my neural network consists of 2 cnn which both are followed by max-pooling and a linear layer that has 2 outputs, one for x and the other one for y. The input is a 720x720 image.

class MyNeuralNetwork(torch.nn.Module):
    def __init__(self):
        super(MyNeuralNetwork, self).__init__()
        self.conv1 = torch.nn.Conv2d(4, 5, 5)
        self.conv2 = torch.nn.Conv2d(5, 5, 5)
        self.pool = torch.nn.MaxPool2d(3, 3)
        self.linear = torch.nn.Linear(5 * 78 * 78, 2)

    def forward(self, x):
        x = self.conv1(x)
        x = self.pool(x)
        x = self.conv2(x)
        x = self.pool(x)
        x = x.view(x.size(0), -1)
        x = self.linear(x)
    return x

I have the pathnames of the images saved in a csv file. the x and y coordinates are saved in a different csv file. Here is the code for my Dataset.

class MyHand(Dataset):
 """Creating the proper dataset to feed my neural network"""
    def __init__(self, name_path, root_dir, results_path, transform=None):
       self.names = pd.read_csv(name_path)
       self.rootdir = root_dir
       self.transform = transform
       self.results = pd.read_csv(results_path)

    def __len__(self):
        length = len(self.names.columns)
        return length

    def __getitem__(self, index):
        img_path = os.path.join(self.rootdir, self.names.columns[index])
        image = pl.imread(img_path)
        x_top_left_corner = torch.tensor(self.results.iloc[index, 0])
        y_top_left_corner = torch.tensor(self.results.iloc[index, 1])
        width = torch.tensor(self.results.iloc[index, 2])
        height = torch.tensor(self.results.iloc[index, 3])


        # calculating the x and y center of my palm
        x_center = x_top_left_corner + width/2
        y_center = y_top_left_corner - height/2

        if self.transform:
            image = self.transform(image)

        return image, x_center, y_center

and the code for training the network is

dataset = MyHand(name_path='path to the names of the images csv', 
results_path='path to the results cvs', 
transform=torchvision.transforms.ToTensor( ))
loader = DataLoader(dataset=dataset, batch_size=4)
model = MyNeuralNetwork()
criterion = torch.nn.MSELoss()
EPOCHS = 5
LEARNING_RATE = 0.001
optimizer = optim.SGD(model.parameters(), LEARNING_RATE)

for epoch in range(EPOCHS):
    print("epoch:", epoch)
    for data in dataset:
        pic, x, y = data
        model.zero_grad()
        outpout = model(pic[None, :, :, :])
        loss1 = criterion(outpout[0, 0], x)
        loss2 = criterion(outpout[0, 1], y)
        loss = loss1 + loss2
        loss.backward()
        print(loss)

but as you can see below my loss function has exactly the same results at each epoch and it doesn't decrease at all. What can i do for that? I tried different values of learning rate but still the same.

user_8909353 · Answer

Your loss values are extremly high as you see. I would propose that you normalize your outputs by using the sigmoid activation function. Now the coordinates are in the range 0-1 and can be later translated to the image by multiplying them with 720. To calculate the loss, you have to divide your target cooridnates by 720. Then you should get a nice and stable loss in the range 0-1. Also:

either decay your learning rate or try a smaller one
scale your image down (I don't know how the images look like but 720x720 is quite big)
use three convolutions with smaller kernels
add a second linear layer

My Loss Function doesn't get smaller values during training

1 Answers