I'm trying to follow this pytorch's optim documentation on using optimizer.param_groups to group several learning rate schedulers, but it seems like it only works for the LambdaLR one.
If I have my model:
net = Model()
I naively tried to add the schedulers with:
learning_rate = 0.02
optimizer = optim.Adam(net.parameters(), lr = learning_rate)
net.optimizer = optimizer
scheduler1 = torch.optim.lr_scheduler.CyclicLR(net.optimizer, base_lr=1e-10, max_lr=1e-3, step_size_up=3000, mode="triangular2", cycle_momentum=False)
net.scheduler1 = scheduler1
scheduler2 = torch.optim.lr_scheduler.CyclicLR(optimizer, base_lr=1e3, max_lr=1e-1, step_size_up=1000, mode="triangular2", cycle_momentum=False)
net.scheduler2 = scheduler2
I'm trying to use scheduler1 to train one of my neural networks, and scheduler2 is just used for its value. That is, at each epoch I'm using its value to plug in into some function:
for epoch in num_epochs:
loss = 5 - optimizer.param_groups[0]["lr"]
Note that that's just an example function.
The issue is that I think the model just takes the scheduler2 as its scheduler. Since
optimizer.param_groups[1]["lr"] returned:
IndexError: list index out of range
Update 1
For clarification, I have a neural network inside my model net. I want this neural network to have 1 lr scheduler, scheduler1. I have a certain function inside my model net, that needs to take certain values at each epoch. The values that it needs to take resemble the values from scheduler2. Hence, I need to add both schedulers to my model, using the first for my neural network, and the second to my function (not relating to any model's parameters except for the epoch number).