adding position and segment encoding directly to the token attention matrix per-head

Viewed 8

I need the python code that implement the positional encoding to attention matrix per head as proposed by google research "A Simple and Effective Positional Encoding for Transformers". I have seached a lot but I couldnt find that add positional encoding to the attention matrix, al of them implemented it in input embedding.For example in this code """https://github.com/guocheng2018/Transformer-Encoder/tree/master/transformer_encoder""" we have positional encoding in input.My question is how do I change this python code to what I explained.Thanks

0 Answers
Related