Identifying the word picked by hugging face pipeline fill-mask

user_3472360 0
Viewed 548

I want to use hugging face's fill-mask pipeline to guess a masked token and then extract just the guessed token as a word. This code should do that:

!pip install -q transformers
model = pipeline('fill-mask')
outcome = model("Kubernetes is a container orchestration <mask>")[0]

#Prints: "Kubernetes is a container orchestration platform" 
print(outcome['sequence']) 

token = outcome['token'] 

#Prints: 1761
print(token)

#Prints: Ġplatform 
print(model.tokenizer.convert_ids_to_tokens(token))

But I am finding that it gives me back "Ġplatform" instead of "platform" - does anyone know why this is or what can be going on here?

huggingface-transformers neural-network nlp python
2 Answers

This is simply a peculiarity of the underlying model (see here to check that this is distilroberta-base).
Specifically, distilled models use the same tokenizer as their "teacher models" (in this case, RoBERTa). RoBERTa, in turn, has a tokenizer that is working strictly without any form of whitespaces, see also this thread on OpenAI's GPT-2 model, which is using the same tokenization strategy (see here).

Specifically, you can note that it is always the same unicode character \u0120 that denotes the start of a new word. Comparatively, words that would consist of multiple subwords would have no such starting characters for the later subwords.

I.e., complication would be split into two fictionary subwords Ġcompli cation.

Therefore, you can simply drop the Ġ if it appears in the word.

user_36072030

As of 2021-08-14, the output of the pipeline contains all the information that you requested. Note that token_str may contain spaces.

from transformers import pipeline

unmasker = pipeline("fill-mask")

s="Excuse me sir, <mask> you speak English?"

unmasker(s,top_k=5)

[{'score': 0.9028477668762207,
  'sequence': 'Excuse me sir, do you speak English?',
  'token': 109,
  'token_str': ' do'},
 {'score': 0.05048234760761261,
  'sequence': 'Excuse me sir, did you speak English?',
  'token': 222,
  'token_str': ' did'},
 {'score': 0.015572326257824898,
  'sequence': 'Excuse me sir, can you speak English?',
  'token': 64,
  'token_str': ' can'},
 {'score': 0.007022920995950699,
  'sequence': 'Excuse me sir, Do you speak English?',
  'token': 1832,
  'token_str': ' Do'},
 {'score': 0.005521782673895359,
  'sequence': 'Excuse me sir, if you speak English?',
  'token': 114,
  'token_str': ' if'}]
user_47522230
Related