Set negative learning rate in torch.optim. One way fails while another works, why?

Viewed 29

I want to define an optimizer in pytorch for my code.

The code needs to have a negative learning rate for a special alogrithm.

When I write code like this:

import torch

class Net(torch.nn.Module):
    def __init__(self):
        super().__init__()
        self.linear1 = torch.nn.Linear(3, 2)

    def forward(self, x):
        return self.linear1(x)

if __name__ == '__main__':
    net = Net()
    main_model_params = [{
        "params": [p for _, p in net.named_parameters()],
        "weight_decay": 0.0,
    }]
    optimizer = torch.optim.AdamW(
                main_model_params,
                lr = -0.1
            )
    print(optimizer)

Then an error will happens:

Traceback (most recent call last):
  File ".\experiment\optim_test.py", line 18, in <module>
    optimizer = torch.optim.AdamW(
  File "C:\ProgramData\Miniconda3\lib\site-packages\torch\optim\adamw.py", line 70, in __init__
    raise ValueError("Invalid learning rate: {}".format(lr))
ValueError: Invalid learning rate: -0.1

But when I change a method to write the code:

import torch
import torch.nn

class Net(torch.nn.Module):
    def __init__(self):
        super().__init__()
        self.linear1 = torch.nn.Linear(3, 2)

    def forward(self, x):
        return self.linear1(x)

if __name__ == '__main__':
    net = Net()
    main_model_params = [{
        "params": [p for _, p in net.named_parameters()],
        "weight_decay": 0.0,
        "lr": -0.1
    }]
    optimizer = torch.optim.AdamW(
                main_model_params,
            )
    print(optimizer)

It can be executed rightly:

AdamW (
Parameter Group 0
    amsgrad: False
    betas: (0.9, 0.999)
    eps: 1e-08
    lr: -0.1
    maximize: False
    weight_decay: 0.0
)

Who can tell me why?

1 Answers

This seems to be an implementation loophole, since the sanity check to make sure the learning is not negative is present in the initializer of the optim.AdamW class here but is not present when looping over the different parameter group parameters once in the parent Optimizer class here. This is the reason why you can provide a negative learning rate when defined inside the group's parameter.

Related