Say I created a Pytorch model:
class Model(nn.Module):
def __init__(self):
super(Model, self).__init__()
self.model_1_encoder = RobertaModel.from_pretrained('roberta-base')
self.model_2_encoder = RobertaModel.from_pretrained('roberta-base')
self.dropout = nn.Dropout(0.5)
self.linear_model_1_out = nn.Linear(768, 512)
self.linear_model_2_out = nn.Linear(768, 512)
self.linear3 = nn.Linear(512, 512)
self.linear4 = nn.Linear(512, 1)
def forward(self, x):
question_input_ids, attn_mask, xyz_input_ids, xyz_attention_mask = torch.tensor(x['input_ids']), torch.tensor(x['attention_mask']), torch.tensor(x['xyz_ids']), torch.tensor(x['xyz_mask'])
model_1_outputs = self.model_1_encoder(question_input_ids, attention_mask=attn_mask)
model_2_outputs = self.model_2_encoder(xyz_input_ids, attention_mask = xyz_attention_mask)
model_1_outputs = self.dropout(model_1_outputs[0])
model_2_outputs = self.dropout(model_2_outputs[0])
model_2_outputs = torch.sum(model_2_outputs, dim=0)
model_2_outputs = torch.unsqueeze(model_2_outputs, 0)
model_2_outputs = model_2_outputs.repeat(model_1_outputs.shape[0], 1, 1)
model_1_outputs = self.linear_model_1_out(model_1_outputs)
model_2_outputs = self.linear_model_2_out(model_2_outputs)
model_1_and_xyz_sum = model_2_outputs + model_1_outputs
outputs = self.dropout(model_1_and_xyz_sum)
outputs = outputs[:,0,:].view(-1,512)
outputs = torch.relu(self.linear3(outputs))
outputs = self.dropout(outputs)
outputs = torch.relu(self.linear4(outputs))
return outputs
model = Model()
From this SO question I understand that I need to create a config file for the Pytorch model that will be loaded by the model. They referenced How to add a model to Transformers?, which if I understands correctly requires manual writing of each layer in my custom model in addition to various other inputs and steps. I also found this question on their forum, which referenced this, which had even more steps to complete in order to convert the Pytorch model.
How can I automatically convert my Pytorch model to a Hugginface model so that I can use their Trainer class (in addition to other functionalities like the DataCollator, etc.)?
Potential optimal pseudocode:
my_pytorch_model = Model()
my_huggingface_model = Custom_huggingface_Model.load_model(my_pytorch_model)