How to add LSTM layer on top of Huggingface BERT model

user_15025011 0
Viewed 2840

I am working on a binary classification task and would like to try adding lstm layer on top of the last hidden layer of huggingface BERT model, however, I couldn't reach the last hidden layer. Is it possible to combine BERT with LSTM?

tokenizer = BertTokenizer.from_pretrained(model_path)
tain_inputs, train_labels, train_masks = data_prepare_BERT(
    train_file, lab2ind, tokenizer, content_col, label_col, 
    max_seq_length)
validation_inputs, validation_labels, validation_masks = data_prepare_BERT(
    dev_file, lab2ind, tokenizer, content_col, label_col,max_seq_length)

# Load BertForSequenceClassification, the pretrained BERT model with a single linear classification layer on top.
model = BertForSequenceClassification.from_pretrained(
    model_path, num_labels=len(lab2ind))
recurrent-neural-network bert-language-model huggingface-transformers pytorch nlp
1 Answers

Indeed it is possible, but you need to implement it yourself. BertForSequenceClassification class is a wrapper for BertModel. It runs the model, takes the hidden state corresponding to the [CLS] tokens, and applies a classifier on top of that.

In your case, you can the class as a starting point, and add there an LSTM layer between the BertModel and the classifier. The BertModel returns both the hidden states and a pooled state for classification in a tuple. Just take the other tuple member than is used in the original class.

Although it is technically possible, I would expect any performance gain compared to using BertForSequenceClassification. Finetuning of the Transformer layers can learn anything that an additional LSTM layer is capable of.

user_56523130
Related