Shall we lower case input data for (pre) training a BERT uncased model using huggingface?

Viewed 3351
2 Answers

Tokenizer will take care of that.

A simple example:

import torch
from transformers import BertTokenizer
tokenizer = BertTokenizer.from_pretrained('bert-base-uncased', max_length = 10, padding_side = 'right')

input_ids = torch.tensor(tokenizer.encode('this is a cat', add_special_tokens=True, max_length = 10, pad_to_max_length = True)).unsqueeze(0)
print(input_ids)

input_ids = torch.tensor(tokenizer.encode('This is a Cat', add_special_tokens=True, max_length = 10, pad_to_max_length = True)).unsqueeze(0)
print(input_ids)

Out:

tensor([[ 101, 2023, 2003, 1037, 4937,  102,    0,    0,    0,    0]])
tensor([[ 101, 2023, 2003, 1037, 4937,  102,    0,    0,    0,    0]])

But in case of cased,

tokenizer = BertTokenizer.from_pretrained('bert-base-cased', max_length = 10, padding_side = 'right')

input_ids = torch.tensor(tokenizer.encode('this is a cat', add_special_tokens=True, max_length = 10, pad_to_max_length = True)).unsqueeze(0)
print(input_ids)

input_ids = torch.tensor(tokenizer.encode('This is a Cat', add_special_tokens=True, max_length = 10, pad_to_max_length = True)).unsqueeze(0)
print(input_ids)
tensor([[ 101, 1142, 1110,  170, 5855,  102,    0,    0,    0,    0]])

tensor([[ 101, 1188, 1110,  170, 8572,  102,    0,    0,    0,    0]])

I think the bert-base-uncased model will lower case the text irrespective of what you pass to the model. You can also try playing with a toy dataset and print the tokens using the BERT tokenizer so as to just confirm.

Related