I have a DNN class in pytorch. It inputs two featuers vectors and use two different fully connected layers to feedforward in the first few layers. Later, the two layers catenate to one single output. How can I set different regularization parameters for the last layer regarding the first and second feature? Here is the class:
class DNN(nn.Module):
def __init__(self,):
super(DNN, self).__init__()
self.activation = nn.ReLU()
self.fc1 = nn.ModuleList([nn.Sequential(
nn.Linear(10, 4),self.activation,
)])
self.fc2 = nn.ModuleList([nn.Sequential(
nn.Linear(10, 4),self.activation,
)])
self.out= nn.Linear(8, 1)
def forward(self,X1,X2):
for layer in self.fc1:
X1=layer(X1)
for layer in self.fc2:
X2=layer(X2)
X=torch.cat((X1,X2),dim=1)
y=self.out(X)
return y
For the optimizer, I know how to set up different l2 for parameters at different layers as the following. \
optim_params=[{'params': self.fc1.parameters(), 'weight_decay': 0.01},
{'params': self.fc1.parameters(), 'weight_decay': 0.001},
{'params': self.out.parameters(), 'weight_decay': 0.001},
]
optimizer = optim.Adam(optim_params)
But any possible to set l2 different for self.out.parameter() at different position?