How to customize the positional embedding?

user_2195440 0
Viewed 163

I am using the Transformer model from Hugging face for machine translation. However, my input data has relational information as shown below:

enter image description here

I want to craft a graph like the like the following:

 ________
 |       |
 |      \|/
He ended his meeting on Tuesday night.
/|\ |         |          /|\
 |  |         |           | 
 |__|         |___________|

Essentially each token in the sentence is a node and there could be an edge embedded between the tokens.

In a normal transformer, the tokens are processed into token embeddings, also there is an encoding of each position which resulted into positional embeddings.

How could I do something similar with the edge information?

Theoretically I could take the edge type and the positional encoding of a node and output an embedding. The embeddings of all the edges can be added to the positional embeddings for the corresponding nodes.

Ideally, I would like to implement this with the hugging face transformer.

I am struggling to understand how could I update the positional embedding here:

https://github.com/huggingface/transformers/blob/main/src/transformers/models/longformer/modeling_longformer.py#L453

        self.position_embeddings = nn.Embedding(
            config.max_position_embeddings, config.hidden_size, padding_idx=self.padding_idx
        )
huggingface transformer-model huggingface-tokenizers huggingface-transformers deep-learning
1 Answers

Your question is generally about how to use linguistic representations to enrich word representations in Transformer language models. This is an open research question with no clear answer. One option given by Prange et al. (2022) is to define a function which returns a subgraph of the whole-sentence graph given a particular token, and to process the token's subgraph so that it is represented in a fixed-length vector which can then be readily combined with the token's representation for use in the rest of the Transformer LM. Their related work section reviews other approaches and is worth a look.

user_45399000
Related