Fine tune GPT-2 on large text for generate a domain text

user_4048691 0
Viewed 3253

Tryin to train GPT-2 on a very large text, in order to generate text from specific domain.
Working with tensorflow2 .

For example, let's say I have all of Harry Potter books :)
And I want to train the GPT-2 on them, so I could later generate text from the Harry Potter domain.

from tensorflow.keras.utils import get_file
from transformers import GPT2Tokenizer, TFGPT2Model

text = '...'
# Length of text: 474429 characters
# 84 unique characters

tokenizer = GPT2Tokenizer.from_pretrained('gpt2-medium')
model = TFGPT2Model.from_pretrained('gpt2-medium')

encoded_input = tokenizer(text, return_tensors='tf') # ERROR
output = model(encoded_input)

input_ids = tokenizer.encode('severus snape', return_tensors='tf')
greedy_output = model.generate(input_ids, max_length=50)
print(tokenizer.decode(greedy_output[0], skip_special_tokens=True))

ERROR: Token indices sequence length is longer than the specified maximum sequence length for this model (149887 > 1024). Running this sequence through the model will result in indexing errors

So how would I make it work?
How to feed the model a large new text to train on?

EDIT:
when Trying to concat, tokenizer works, but model doesn't:

from textwrap import wrap
text_batches = wrap(text, 1000)

encoded_input = None

for tb in text_batches:
    current = tokenizer(tb, return_tensors='tf')
  
    if encoded_input == None:
        encoded_input = current
    else:
        encoded_input['input_ids']      = tf.concat([encoded_input['input_ids'], current['input_ids']], axis=-1)
        encoded_input['attention_mask'] = tf.concat([encoded_input['attention_mask'], current['attention_mask']], axis=1)

output = model(encoded_input) # ERROR

ERROR: InvalidArgumentError: indices[0,1024] = 1024 is not in [0, 1024) [Op:ResourceGather]

What am I missing?

huggingface-transformers keras tensorflow deep-learning nlp
1 Answers

Your problem is not related to training on different domains. Rather, you're simply providing a text length (apparently 149887 tokens) that's longer than the maximum length that the model can support (1024). You have three options:

  1. Manually truncate your input strings to the max length of tokens.

  2. Set the max_length parameter in the call to your tokenizer, e.g. tokenizer(text, max_length=1024, ...). Be sure to read all the available options for the Tokenizer class.

  3. Revisit why you need a text string of 149K tokens. Is this the whole body of the text? Should you instead use sentences?

user_45613140
Related