Huggingface Transformers Tensorflow fine-tuned distilgpt2 bad outputs

I fine-tuned a model starting from the 'distilgpt2' checkpoint. I fit the model with the model.fit() method and saved the resulting model with the .save_pretrained() method.

When I use this model to generate text:

import transformers
from transformers import TFAutoModelForCausalLM, AutoTokenizer

original_model = 'distilgpt2'
path2model = 'clm_model_save'
path2tok = 'clm_tokenizer_save'

tuned_model = TFAutoModelForCausalLM.from_pretrained(path2model, from_pt=False)
tuned_tokenizer = AutoTokenizer.from_pretrained(path2tok)

input_context = 'The dog'
input_ids = tuned_tokenizer.encode(input_context, return_tensors='tf')  # encode input context

outputs = tuned_model.generate(input_ids=input_ids, 
                               max_length=40, 
                               temperature=0.7, 
                               num_return_sequences=3, 
                               do_sample=True)  # generate 3 candidates using sampling

for i in range(3): #  3 output sequences were generated
    print(f'Generated {i}: {tuned_tokenizer.decode(outputs[i], skip_special_tokens=True)}')

The model returns the output:

>>>All model checkpoint layers were used when initializing TFGPT2LMHeadModel.

>>>All the layers of TFGPT2LMHeadModel were initialized from the model checkpoint at clm_model_save.
>>>If your task is similar to the task the model of the checkpoint was trained on, you can already use TFGPT2LMHeadModel for predictions without further training.
>>>Setting `pad_token_id` to 50256 (first `eos_token_id`) to generate sequence

>>>Generated 0: The dog!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
>>>Generated 1: The dog!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
>>>Generated 2: The dog!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

When I use the original checkpoint, distilgpt2, the model generates text just fine. Is this a sign of some sort of misconfiguration? Or is this simply a sign of a poorly trained model?

I've tried using the original checkpoint tokenizer, manually setting the pad_token_id, using a much longer input context, and changing several parameters for the .generate() method. Same results each time.

Also, I added special tokens to my tuned_tokenizer:

tuned_tokenizer.special_tokens_map

>>>{'bos_token': '<|startoftext|>',
>>> 'eos_token': '<|endoftext|>',
>>> 'unk_token': '<|endoftext|>',
>>> 'pad_token': '<|PAD|>'}

Compared to the original tokenizer:

tokenizer.special_tokens_map

>>> {'bos_token': '<|endoftext|>',
>>> 'eos_token': '<|endoftext|>',
>>> 'unk_token': '<|endoftext|>'}