tgt and src have to have equal features for a Transformer Network in Pytorch

Viewed 850

I am attempting to train EEG data through a transformer network. The input dimensions are 50x16684x60 (seq x batch x features) and the output is 16684x2. Right now I am simply trying to run a basic transformer, and I keep getting an error telling me

RuntimeError: the feature number of src and tgt must be equal to d_model

Why would the source and target feature number ever be equal? Is it possible to run such a dataset through a transformer?

Here is my basic model:

input_size = 60 # seq x batch x features
hidden_size = 32
num_classes = 2
learning_rate = 0.001
batch_size = 64
num_epochs = 2
sequence_length = 50
num_layers = 2
dropout = 0.5

class Transformer(nn.Module):
    def __init__(self, input_size, hidden_size, num_layers, num_classes):
        super(Transformer, self).__init__()
        self.hidden_size = hidden_size
        self.num_layers = num_layers
        
        self.transformer = nn.Transformer(60, 2)
        self.fc = nn.Linear(hidden_size * sequence_length, num_classes)
    
    def forward(self, x, y):
        
        # Forward Propogation
        out, _ = self.transformer(x,y)
        out = out.reshape(out.shape[0], -1)
        out = self.fc(out)
        return out

model = Transformer(input_size, hidden_size, num_layers, num_classes)

criterion = nn.MSELoss()
optimizer = optim.Adam(model.parameters(), lr=learning_rate)

for epoch in range(num_epochs):
    for index in tqdm(range(16684)):
        X, y = (X_train[index], Y_train[index])
        print(X.shape, y.shape)
    
        output = model(X, y)

        loss = criterion(output, y)
        
        model.zero_grad()
        loss.backward()
        
        optimizer.step()
        
        if index % 500 == 0:
            print(f"Epoch {epoch}, Batch: {index}, Loss: {loss}")

0 Answers
Related