BERT model : "enable_padding() got an unexpected keyword argument 'max_length'"

user_9621262 0
Viewed 2219

I am trying to implement the BERT model architecture using Hugging Face and KERAS. I am learning this from the Kaggle (link) and try to understand it. When I tokenized my data, I face some problems and get an error message. The error msg is:

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-20-888a40c0160b> in <module>
----> 1 x_train = fast_encode(train1.comment_text.astype(str), fast_tokenizer, maxlen=MAX_LEN)
      2 x_valid = fast_encode(valid.comment_text.astype(str), fast_tokenizer, maxlen=MAX_LEN)
      3 x_test = fast_encode(test.content.astype(str), fast_tokenizer, maxlen=MAX_LEN )
      4 y_train = train1.toxic.values
      5 y_valid = valid.toxic.values

<ipython-input-8-de591bf0a0b9> in fast_encode(texts, tokenizer, chunk_size, maxlen)
      4     """
      5     tokenizer.enable_truncation(max_length=maxlen)
----> 6     tokenizer.enable_padding(max_length=maxlen)
      7     all_ids = []
      8 

TypeError: enable_padding() got an unexpected keyword argument 'max_length'

and the code is:

x_train = fast_encode(train1.comment_text.astype(str), fast_tokenizer, maxlen=192)
x_valid = fast_encode(valid.comment_text.astype(str), fast_tokenizer, maxlen=192)
x_test = fast_encode(test.content.astype(str), fast_tokenizer, maxlen=192 )
y_train = train1.toxic.values
y_valid = valid.toxic.values

and the function fast_encode is here:

def fast_encode(texts, tokenizer, chunk_size=256, maxlen=512):
    """
    Encoder for encoding the text into sequence of integers for BERT Input
    """
    tokenizer.enable_truncation(max_length=maxlen)
    tokenizer.enable_padding(max_length=maxlen)
    all_ids = []
    
    for i in tqdm(range(0, len(texts), chunk_size)):
        text_chunk = texts[i:i+chunk_size].tolist()
        encs = tokenizer.encode_batch(text_chunk)
        all_ids.extend([enc.ids for enc in encs])
    
    return np.array(all_ids)

What should I do now?

bert-language-model padding tokenize nlp
1 Answers

The tokenizer used here is not the regular tokenizer, but the fast tokenizer provided by an older version of the Huggingface tokenizer library.

If you wish to create the fast tokenizer using the older version of huggingface transformers from the notebook, you can do this:

from tokenizers import BertWordPieceTokenizer

# First load the real tokenizer
tokenizer = transformers.DistilBertTokenizer.from_pretrained('distilbert-base-multilingual-cased')
# Save the loaded tokenizer locally
tokenizer.save_pretrained('.')
# Reload it with the huggingface tokenizers library
fast_tokenizer = BertWordPieceTokenizer('vocab.txt', lowercase=False)
fast_tokenizer

However, the process of using a fast tokenizer has become significantly simpler since I wrote this code. If you look at the Proprocessing data tutorial by Huggingface, you will notice that you simply need to do:

tokenizer = AutoTokenizer.from_pretrained('bert-base-cased')

batch_sentences = [
    "Hello world",
    "Some slightly longer sentence to trigger padding"
]
batch = tokenizer(batch_sentences, padding=True, truncation=True, return_tensors="tf")

This is because the fast tokenizer (which is written in Rust) is automatically used whenever it's available.

user_138370910
Related