I want to define an optimizer in pytorch for my code.
The code needs to have a negative learning rate for a special alogrithm.
When I write code like this:
import torch
class Net(torch.nn.Module):
def __init__(self):
super().__init__()
self.linear1 = torch.nn.Linear(3, 2)
def forward(self, x):
return self.linear1(x)
if __name__ == '__main__':
net = Net()
main_model_params = [{
"params": [p for _, p in net.named_parameters()],
"weight_decay": 0.0,
}]
optimizer = torch.optim.AdamW(
main_model_params,
lr = -0.1
)
print(optimizer)
Then an error will happens:
Traceback (most recent call last):
File ".\experiment\optim_test.py", line 18, in <module>
optimizer = torch.optim.AdamW(
File "C:\ProgramData\Miniconda3\lib\site-packages\torch\optim\adamw.py", line 70, in __init__
raise ValueError("Invalid learning rate: {}".format(lr))
ValueError: Invalid learning rate: -0.1
But when I change a method to write the code:
import torch
import torch.nn
class Net(torch.nn.Module):
def __init__(self):
super().__init__()
self.linear1 = torch.nn.Linear(3, 2)
def forward(self, x):
return self.linear1(x)
if __name__ == '__main__':
net = Net()
main_model_params = [{
"params": [p for _, p in net.named_parameters()],
"weight_decay": 0.0,
"lr": -0.1
}]
optimizer = torch.optim.AdamW(
main_model_params,
)
print(optimizer)
It can be executed rightly:
AdamW (
Parameter Group 0
amsgrad: False
betas: (0.9, 0.999)
eps: 1e-08
lr: -0.1
maximize: False
weight_decay: 0.0
)
Who can tell me why?