PyTorch: defined layer was not involved in the forward propagation but influenced the loss value

user_14466958 0
Viewed 151

Recently, I met a confusing phenomenon when I used Pytorch to do simple experiments on logistic regression.

The question is when I fixed the random seed like this:

def set_seed(seed, cuda=True):
    np.random.seed(seed)
    torch.manual_seed(seed)
    if cuda:
        torch.cuda.manual_seed(seed)

and defined the following model with 2 layers:

class net(nn.Module):
    def __init__(self):
        super(net, self).__init__()
        self.hidden = nn.Linear(784, 100)
        self.output = nn.Linear(100, 10)

    def forward(self, x):
        x = self.hidden(x)
        x = self.output(x)
        return x

trained the network with:

criterion = nn.CrossEntropyLoss()
optimizer = torch.optim.SGD(net.parameters(), lr=0.1)

The orignal loss value was 0.6422, which is reproducible.

However, when I added an additional layer that was not involoved in the forward process like this:

class net(nn.Module):
    def __init__(self):
        super(net, self).__init__()
        self.hidden = nn.Linear(784, 100)
        self.output = nn.Linear(100, 10)
        self.add = nn.Linear(10,10)

    def forward(self, x):
        x = self.hidden(x)
        x = self.output(x)
        return x

The original loss value changed to 0.7431, which was not equal to the previous one and the model performance dropped simultaneously.

I really wonder the reason to this. Thank you!

pytorch deep-learning python
1 Answers

This is completely expected if there are other sources of randomness (something that consumes the RNG) before computing the loss. As you didn't provide a Minimal, Reproducible Example, I'd guess that you're using a DataLoader with shuffle=True. In this case, even though you do not use the self.add layer, when you initialize it, it consumes the RNG; therefore leading to a different order to the samples. If the randomness is coming from a DataLoader with shuffle=True, you can control that by providing a different RNG to the DataLoader. Something like this:

import numpy as np

import torch
from torch import nn

import torchvision
from torchvision.transforms import ToTensor

def set_seed(seed, cuda=True):
    np.random.seed(seed)
    torch.manual_seed(seed)
    if cuda:
        torch.cuda.manual_seed(seed)

class net(nn.Module):
    def __init__(self):
        super(net, self).__init__()
        self.hidden = nn.Linear(784, 100)
        self.output = nn.Linear(100, 10)
        # self.add = nn.Linear(10, 10)  # try with and without
    def forward(self, x):
        x = self.hidden(x)
        x = self.output(x)
        return x

set_seed(0)

m = net()

bs = 4
ds = torchvision.datasets.MNIST(root=".", train=True, transform=ToTensor(), download=True)
rng_dl = torch.Generator()
dl = torch.utils.data.DataLoader(ds, batch_size=bs, shuffle=True, num_workers=0, generator=rng_dl)

criterion = nn.CrossEntropyLoss()
optimizer = torch.optim.SGD(m.parameters(), lr=0.1)

for x, y in dl:
    y_hat = m(x.view(bs, -1))
    l = criterion(y_hat, y)
    print(l)
    exit()

Keep in mind that it could be several other things, such as data augmentation and other calls to functions that rely on random ops. If you can provide an MRE, I could try and give a more specific answer.

user_42282750
Related