How to fine tune a masked language model?

Question

I'm trying to follow the huggingface tutorial on fine tuning a masked language model (masking a set of words randomly and predicting them). But they assume that the dataset is in their system (can load it with from datasets import load_dataset; load_dataset("dataset_name")). However, my input dataset is a long string:

text = "This is an attempt of a great example. "
dataset = text * 3000

I followed their approach and tokenized each it:

from transformers import AutoTokenizer
from transformers import AutoModelForMaskedLM
import torch
from transformers import DataCollatorForLanguageModeling

model_checkpoint = "distilbert-base-uncased"
model = AutoModelForMaskedLM.from_pretrained(model_checkpoint)
tokenizer = AutoTokenizer.from_pretrained(model_checkpoint)

def tokenize_long_text(tokenizer, long_text):
    individual_sentences = long_text.split('.')
    tokenized_sentences_list = tokenizer(individual_sentences)['input_ids']
    tokenized_sequence = [x for xs in tokenized_sentences_list for x in xs]
    return tokenized_sequence 

tokenized_sequence = tokenize_long_text(tokenizer, long_text)

Following by chunking it into equal length segments:

def chunk_long_tokenized_text(tokenizer_text, chunk_size):
    # Compute length of long tokenized texts
    total_length = len(tokenizer_text)
    # We drop the last chunk if it's smaller than chunk_size
    total_length = (total_length // chunk_size) * chunk_size
    
    return [tokenizer_text[i : i + chunk_size] for i in range(0, total_length, chunk_size)]

chunked_sequence = chunk_long_tokenized_text(tokenized_sequence, 30)

Created a data collator for random masking:

data_collator = DataCollatorForLanguageModeling(tokenizer=tokenizer, mlm_probability=0.15) # expects a list of dicts, where each dict represents a single chunk of contiguous text

Example of how it works:

d = {}
d['input_ids'] = chunked_sequence[0]
d

>>>{'input_ids': [101,
  2023,
  2003,
  1037,
  2307,
  103,...

for chunk in data_collator([ d ])["input_ids"]:
    print(f"
'>>> {tokenizer.decode(chunk)}'")
>>>'>>> [CLS] this is a great [MASK] [SEP] [CLS] this is a great [MASK] [SEP] [CLS] this is a great [MASK] [SEP] [CLS] this is a great [MASK] [SEP] [CLS] this'

However, the remaining steps (which I believe is just the training component) seem to only work using their trainer method, which can only take their dataset.

How can this work with a dataset in the form of a string?

How to fine tune a masked language model?

0 Answers