pooled output vs sequence output for NER with BERT

user_6017833 0
Viewed 755

It makes sense that pooled output is used for sentence level analysis (e.g classification). I am wondering, is the token level embedding better for named entity recongition? I would've thought so, but am unsure how to take the sequence output and manipulate for NER. Do I just take a slice [:, :, -1] for [batch_size, num_tokens, 768]?

nlp python
1 Answers

If we talk about bert, there we get two output

o1, o2 = self.bert(ids, attention_mask=mask)

  • o1-Sequential output: Each and every token will receive its own embedding (can be used in NER Pooled output.)
  • o2-Pooled output: Whole Sentence receives an embedding. (can be used in classification problem )

In the forward pass, you can further modify the bert output according to your use case.

user_83097150
Related