Many-to-many RNN with variable output size in PyTorch

I am trying to implement a many-to-many RNN network. I have defined the model and forward prop as below:

class ManyToManyRNN(nn.Module):
    def __init__(self, hid_size=100, output_dim):
        super(ManyToManyRNN, self).__init__()
        self.hidden_size = hid_size
        self.output_dim = output_dim
        self.rnn = nn.LSTM(input_size=4, hidden_size=self.hidden_size, num_layers=1, batch_first=True)
        self.fc1 = nn.Linear(self.hidden_size, self.output_dim)

    def forward(self, inputs):
        outputs = []
        rnn_out, _ = self.rnn(inputs)
        output = self.fc1(rnn_out[:, -1])
        return output

I can set the value of output_dim while initializing the class and then the value gets fixed. However, in my application, the value of the output dimension keeps changing. I can't fix the value of output_dim before hand.

My question is, how can one construct a many-to-many RNN with output dimension that is flexible? I am open to changing the architecture of the model to achieve this functionality. Any help is appreciated.