UserWarning: The .grad attribute of a Tensor that is not a leaf Tensor is being accessed

Question

I am creating Logistic Regression in Pytorch from scratch. But I am facing an issue when I am updating trainable parameters Weights & biases. This is my implementation,

class LogisticRegression():
    
    def __init__(self, n_iter, lr):
        self.n_iter = n_iter
        self.lr = lr
    
    def fit(self, dataset):
        device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
        n = next(iter(dataset))[0].shape[1]
        self.w = torch.zeros(n, requires_grad=True).to(device)
        self.b = torch.tensor(0., requires_grad=True).to(device)
        
        for i in range(self.n_iter):
            with tqdm(total=len(dataset)) as pbar:
                for x, y in dataset:
                    x = x.to(device)
                    y = y.to(device)
                    y_pred = self.predict(x.float())
                    loss = self.loss(y, y_pred)
                    loss.backward()
                    with torch.no_grad():
                        print(self.w, self.b)
                        self.w -= self.w.grad * self.lr
                        self.b -= self.b.grad * self.lr
                        self.w.grad.zero_()
                        self.b.grad.zero_()
                    pbar.update(1)
            print(f'Epoch: {i} | Loss: {loss}')
    
    def loss(self, y, y_pred):
        y_pred = torch.clip(y_pred, 1e-7, 1 - 1e-7)
        return -torch.mean(
                y * torch.log(y_pred + 1e-7) + 
                (1 - y) * torch.log(1 - y_pred + 1e-7),
            axis=0)
    
    def predict(self, x):
        return self.sigmoid(torch.matmul(x, self.w) + self.b)
    
    def sigmoid(self, x):
        return 1/(1 + torch.exp(-x))

As you can see when I am fitting the model with a dataset I am initializing Weights and biases with zeroes and set requires_grad=True so I can access gradients later. I used the sklearn breast cancer dataset,

X, y = load_breast_cancer(return_X_y=True) # load dataset
x_train, x_test, y_train, y_test = train_test_split(X, y, test_size=0.2) # train test split

# convert all numpy arrays to torch tensor
x_train = torch.tensor(x_train)
x_test = torch.tensor(x_test)
y_train = torch.tensor(y_train)
y_test = torch.tensor(y_test)

# Making it a Torch dataset then into DataLoader
train_dataset = torch.utils.data.TensorDataset(x_train, y_train)
train_loader = torch.utils.data.DataLoader(train_dataset, batch_size=32)

test_dataset = torch.utils.data.TensorDataset(x_test, y_test)
test_loader = torch.utils.data.DataLoader(test_dataset, batch_size=32)

log = LogisticRegression(n_iter=10, lr=0.001)
log.fit(train_loader)

As soon as I fit the dataset into Logistic Regression it gives me this error (I have also added one print statement in Logistic regression just before gradient update in which it is clear that it has grad_fn parameter),

tensor([0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.], device='cuda:0', grad_fn=) tensor(0., device='cuda:0', grad_fn=)

TypeError: unsupported operand type(s) for *: 'NoneType' and 'float'

At the start of this Error, it gives this User warning

UserWarning: The .grad attribute of a Tensor that is not a leaf Tensor is being accessed. Its .grad attribute won't be populated during autograd.backward(). If you indeed want the gradient for a non-leaf Tensor, use .retain_grad() on the non-leaf Tensor. If you access the non-leaf Tensor by mistake, make sure you access the leaf Tensor instead. See github.com/pytorch/pytorch/pull/30531 for more informations.

I need help to solve the error so the gradient update and the model trains successfully!

user_18635613 · Answer

Breast cancer dataset features have big range of possible values, from 0.001 to 1000, and big variances too, so it influence gradients (when gradients become too big it leads to instability and later to NaNs). To overcome such dependence it's common practice to normalize data after splitting, for example:

from sklearn import preprocessing
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split 

X, y = load_breast_cancer(return_X_y=True) # load dataset
x_train, x_test, y_train, y_test = train_test_split(X, y, test_size=0.2) # train test split

scaler = preprocessing.StandardScaler().fit(x_train)  # computing mean and variance of train data
x_train = scaler.transform(x_train) # normalizing train data
x_test = scaler.transform(x_test)   # normalizing test data based on statistics of train

So after that everything should be fine.

UserWarning: The .grad attribute of a Tensor that is not a leaf Tensor is being accessed

1 Answers