Why doesn't BertForMaskedLM generate right masked tokens?

user_3943868 0
Viewed 368

I am testing this piece of code:

from transformers import BertTokenizer, BertModel, BertForMaskedLM

tokenizer = BertTokenizer.from_pretrained("hfl/chinese-roberta-wwm-ext")
model = BertForMaskedLM.from_pretrained("hfl/chinese-roberta-wwm-ext")

from transformers import pipeline

def check_model(model, tokenizer):
    fill_mask = pipeline(
        "fill-mask",
        model=model,
        tokenizer=tokenizer
    )
    print('Fill blank: ')
    fill_mask("我喜欢 {nlp.tokenizer.mask_token}.")

    print('Fill blank: ')
    fill_mask("这个品牌的面膜 {nlp.tokenizer.mask_token}.")

print('Check model ...')
check_model(model, tokenizer)

But it prints out this error message:

raceback (most recent call last):
  File "/Users/congminmin/nlp/embedding/transformer/bert_roberta_wwm_test.py", line 21, in <module>
    check_model(model, tokenizer)
  File "/Users/congminmin/nlp/embedding/transformer/bert_roberta_wwm_test.py", line 15, in check_model
    fill_mask("我喜欢 {nlp.tokenizer.mask_token}.")
  File "/Users/congminmin/.venv/wbkg/lib/python3.7/site-packages/transformers/pipelines/fill_mask.py", line 162, in __call__
    self.ensure_exactly_one_mask_token(masked_index.numpy())
  File "/Users/congminmin/.venv/wbkg/lib/python3.7/site-packages/transformers/pipelines/fill_mask.py", line 90, in ensure_exactly_one_mask_token
    f"No mask_token ({self.tokenizer.mask_token}) found on the input",
transformers.pipelines.base.PipelineException: No mask_token ([MASK]) found on the input
transformer-model bert-language-model huggingface-transformers nlp
1 Answers

That is a string formatting issue. Currently when you call:

"这个品牌的面膜 {nlp.tokenizer.mask_token}."

the string you create is:

'这个品牌的面膜 {nlp.tokenizer.mask_token}.'

What you actually want to do is (formated string literals):

f"我喜欢 {fill_mask.tokenizer.mask_token}."

Output:

'我喜欢 [MASK].'

Full example:

from transformers import BertTokenizer, BertModel, BertForMaskedLM

tokenizer = BertTokenizer.from_pretrained("hfl/chinese-roberta-wwm-ext")
model = BertForMaskedLM.from_pretrained("hfl/chinese-roberta-wwm-ext")

from transformers import pipeline

def check_model(model, tokenizer):
    fill_mask = pipeline(
        "fill-mask",
        model=model,
        tokenizer=tokenizer
    )
    print('Fill blank: ')
    print(fill_mask(f"我喜欢 {fill_mask.tokenizer.mask_token}."))

    print('Fill blank: ')
    print(fill_mask(f"这个品牌的面膜 {fill_mask.tokenizer.mask_token}."))

print('Check model ...')
check_model(model, tokenizer)

Output:

Some weights of the model checkpoint at hfl/chinese-roberta-wwm-ext were not used when initializing BertForMaskedLM: ['cls.seq_relationship.bias', 'cls.seq_relationship.weight']
- This IS expected if you are initializing BertForMaskedLM from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
- This IS NOT expected if you are initializing BertForMaskedLM from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
Check model ...
Fill blank: 
[{'sequence': '我 喜 欢 他.', 'score': 0.20969171822071075, 'token': 800, 'token_str': '他'}, {'sequence': '我 喜 欢 你.', 'score': 0.2071659415960312, 'token': 872, 'token_str': '你'}, {'sequence': '我 喜 欢 她.', 'score': 0.13876770436763763, 'token': 1961, 'token_str': '她'}, {'sequence': '我 喜 欢 的.', 'score': 0.07542475312948227, 'token': 4638, 'token_str': '的'}, {'sequence': '我 喜 欢 它.', 'score': 0.05587303638458252, 'token': 2124, 'token_str': '它'}]
Fill blank: 
[{'sequence': '这 个 品 牌 的 面 膜 好.', 'score': 0.15848451852798462, 'token': 1962, 'token_str': '好'}, {'sequence': '这 个 品 牌 的 面 膜..', 'score': 0.12413082271814346, 'token': 119, 'token_str': '.'}, {'sequence': '这 个 品 牌 的 面 膜 呢.', 'score': 0.09926403313875198, 'token': 1450, 'token_str': '呢'}, {'sequence': '这 个 品 牌 的 面 膜 啊.', 'score': 0.06865812838077545, 'token': 1557, 'token_str': '啊'}, {'sequence': '这 个 品 牌 的 面 膜 1.', 'score': 0.061997584998607635, 'token': 122, 'token_str': '1'}]
user_66648720
Related