How to convert tokenizer output to train_dataset which is required by Trainer API in Huggingface Transformers?

user_14142121 0
Viewed 1112

I tried doing tokenisation using documentation of huggingface transformers

from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained('bert-base-cased')
encoded_input = tokenizer(batch_of_sequences)

Pre Trained Tokenizer gives output of dictionary containing three keys which are

encoded_input = {
'input_ids': [[],[],[]],
'token_type_ids': [[],[],[]],
'attention_mask': [[],[],[]]
}

Trainer API requires input of Train & Eval Dataset of type torch.utils.data.Dataset.

How can we use this output to create training dataset required for Trainer API?

huggingface-transformers python
2 Answers

I guess this solves it:

from datasets import Dataset
dataset = Dataset.from_dict(encoded_inputs)

If you want to train the model, you might also want to add the labels. For me:

encoded_inputs["labels"] = encoded_inputs["input_ids"].copy()

did the trick.

user_96719770

There is a huge amount of examples for using Huggingface transformers in combination with Bert model.

But to describe the general training process, you can load your dataset using Dataset class. Then you can encode your data samples in the get_item method using Bert model's tokenizer. When the method is invoked in each iteration, the encoded data are fed into your NN. I use sklearn.model_selection.train_test_split library for splitting train, eval and test subsets out of the basic set.

Following links can be helpful to figure out a general picture of the whole training process: https://huggingface.co/transformers/quickstart.html

https://towardsdatascience.com/text-classification-with-hugging-face-transformers-in-tensorflow-2-without-tears-ee50e4f3e7ed

https://www.curiousily.com/posts/sentiment-analysis-with-bert-and-hugging-face-using-pytorch-and-python/

user_21373780
Related