Why does tokeniser break down words that are present in vocab

user_9176512 0
Viewed 103

In my understanding, what tokeniser does is that, given each word, the tokeniser will break down the word into sub-words only if the word is not present in the tokeniser.get_vocab() :

def checkModel(model):
    tokenizer = AutoTokenizer.from_pretrained(model)

    allList = []
    for word in tokenizer.get_vocab():
        word = word.lower()

        tokens = tokenizer.tokenize(word)
        try:
            if word[0]!='#' and word[0]!='[' and tokens[0] != word:

                allList.append((word, tokens))
                print(word, tokens)
        except:
            continue 
    return allList

checkModel('bert-base-uncased')
# ideally should return an empty list

However, what I have observed is that some models on huggingface will break down words into smaller pieces even if the word is present in the vocab.

checkModel('emilyalsentzer/Bio_ClinicalBERT')

output: 
welles ['well', '##es']
lexington ['le', '##xing', '##ton']
palestinian ['pale', '##st', '##inian']
...
elisabeth ['el', '##isa', '##beth']
alexander ['ale', '##xa', '##nder']
appalachian ['app', '##ala', '##chia', '##n']
mitchell ['mit', '##chel', '##l']
...
  
4630 # tokens in vocab got broken down, not supposed to happen

I have checked a few models of this behaviour, was wondering why is this happening?

huggingface-tokenizers huggingface-transformers python-3.x python
1 Answers

This is a really interesting question, and I am currently wondering whether it should be considered as a bug report on the Huggingface repo.

EDIT: I realized that it is possible to define model-specific tokenization_config.json files to overwrite the default behavior. One example is the bert-base-cased repository, which has the following content for the tokenizer config:

{
  "do_lower_case": false
}

Given that this functionality is available, I think the best option would be to contact the original author of the work and ask them to potentially consider this configuration (if appropriate for the general use case).

Original Answer:

As it turns out, the vocabulary word that you are checking for is welles, yet the vocab file itself only contains Welles. Notice the difference in the uppercased first letter?
It turns out you can manually force the tokenizer to specifically check for cased vocabulary words, in which case it works fine.

from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("emilyalsentzer/Bio_ClinicalBERT",
                                          do_lower_case=False)  # This is different
print(tokenizer.do_lower_case)
# Output: False

# Lowercase input will result in split word
tokenizer.convert_ids_to_tokens(tokenizer("welles")["input_ids"])
# Output: ['[CLS]', 'well', '##es', '[SEP]']

# Uppercase input will correctly *not split* the word
tokenizer2.convert_ids_to_tokens(tokenizer2("Welles")["input_ids"])
['[CLS]', 'Welles', '[SEP]']

Per default, however, this is not the case, and all words will be converted to lowercase, which is why you cannot find the word:

from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("emilyalsentzer/Bio_ClinicalBERT")

# Per default, lowercasing is enabled!
print(tokenizer.do_lower_case)

# Output: True

# This time now we get the same (lowercased) output both times!
tokenizer.convert_ids_to_tokens(tokenizer("welles")["input_ids"])
['[CLS]', 'well', '##es', '[SEP]']
tokenizer.convert_ids_to_tokens(tokenizer("Welles")["input_ids"])
['[CLS]', 'well', '##es', '[SEP]']
user_36072030
Related