Keyword arguments in BERT call function

Question

In the HuggingFace TensorFlow 2.0 BERT library, the documentation states that:

TF 2.0 models accepts two formats as inputs:

having all inputs as keyword arguments (like PyTorch models), or

having all inputs as a list, tuple or dict in the first positional arguments.

I'm trying to use the first of these two to call a BERT model I created:

from transformers import BertTokenizer, TFBertModel
import tensorflow as tf

bert_tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
bert_model = TFBertModel.from_pretrained('bert-base-uncased')

text = ['This is a sentence', 
        'The sky is blue and the grass is green', 
        'More words are here']
labels = [0, 1, 0]
tokenized_text = bert_tokenizer.batch_encode_plus(batch_text_or_text_pairs=text,
                                                  pad_to_max_length=True,
                                                  return_tensors='tf')
dataset = tf.data.Dataset.from_tensor_slices((tokenized_text['input_ids'],
                                              tokenized_text['attention_mask'],
                                              tokenized_text['token_type_ids'],
                                              tf.constant(labels))).batch(3)
sample = next(iter(dataset))

result1 = bert_model(inputs=(sample[0], sample[1], sample[2]))  # works fine
result2 = bert_model(inputs={'input_ids': sample[0], 
                             'attention_mask': sample[1], 
                             'token_type_ids': sample[2]})  # also fine
result3 = bert_model(input_ids=sample[0], 
                     attention_mask=sample[1], 
                     token_type_ids=sample[2])  # raises an error

But when I execute the last line, I get an error:

TypeError: __call__() missing 1 required positional argument: 'inputs'

Could someone please explain how to properly use the keyword argument style of inputs?

user_3607203 · Answer

It seems that internally, they are interpreting the inputs as input_ids, if you do not put more than just a single tensor as the first argument. You can see this in TFBertModel and then looking for TFBertMainLayer's call function.

For me, I get exactly the same results as result1 and result2 if I do the following:

result3 = bert_model(inputs=sample[0], 
                     attention_mask=sample[1], 
                     token_type_ids=sample[2])

Alternatively, you can also just drop the inputs=, works as well.

Keyword arguments in BERT call function

1 Answers