Getting an error when using a custom huggingface bpe tokenizer to train a t5-small

I am trying to build an NMT model using a t5 and Seq2Seq alongside a custom tokenizer. This is the first time I attempt this as well as use a custom tokenizer. I was able to save the tokenizer and load from file without a hitch. My problem is that when I try to train the t5 I am running into an issue but not sure why...

My code for preprocessing the data is

    train_input = [train_dataset_list[idx]["es"] for idx, input in enumerate(train_dataset_list)]
    train_target = [train_dataset_list[idx]["fr"] for idx, output in enumerate(train_dataset_list)]

tokenized_source = tokenizer.encode_batch(train_input)
tokenized_target = tokenizer.encode_batch(train_target)

I use this before training my model:

data_collator = DataCollatorForSeq2Seq(tokenizer=tokenizer, model=model)

training_args = Seq2SeqTrainingArguments(
    output_dir="./results",
    evaluation_strategy="steps",
    eval_steps=100,
    learning_rate=2e-5,
    per_device_train_batch_size=16,
    per_device_eval_batch_size=16,
    weight_decay=0.01,
    save_total_limit=3,
    num_train_epochs=3,
    fp16=True
)

trainer = Seq2SeqTrainer(
    model=model,
    args=training_args,
    train_dataset=tokenized_source, 
    eval_dataset=tokenized_target,
    tokenizer=tokenizer,
    data_collator=data_collator,
)

trainer.train()

But when I run this I get:

AttributeError                            Traceback (most recent call last)
<ipython-input-70-aae2ee102a06> in <module>()
     38 )
     39 
---> 40 trainer.train()

6 frames
/usr/local/lib/python3.7/dist-packages/transformers/data/data_collator.py in __call__(self, features, return_tensors)
    560         if return_tensors is None:
    561             return_tensors = self.return_tensors
--> 562         labels = [feature["labels"] for feature in features] if "labels" in features[0].keys() else None
    563         # We have to pad the labels before calling `tokenizer.pad` as this method won't pad them and needs them of the
    564         # same length to return tensors.

AttributeError: 'tokenizers.Encoding' object has no attribute 'keys'

However, if I simply look at the tokenized_source or tokenized_target I can see that they look as expected:

portion of tokenized_source:

Encoding(num_tokens=25, attributes=[ids, type_ids, tokens, offsets, attention_mask, special_tokens_mask, overflowing]),
 Encoding(num_tokens=16, attributes=[ids, type_ids, tokens, offsets, attention_mask, special_tokens_mask, overflowing]),
 Encoding(num_tokens=56, attributes=[ids, type_ids, tokens, offsets, attention_mask, special_tokens_mask, overflowing]),
 Encoding(num_tokens=42, attributes=[ids, type_ids, tokens, offsets, attention_mask, special_tokens_mask, overflowing]),
 Encoding(num_tokens=2, attributes=[ids, type_ids, tokens, offsets, attention_mask, special_tokens_mask, overflowing])

portion of tokenized_target:

Encoding(num_tokens=23, attributes=[ids, type_ids, tokens, offsets, attention_mask, special_tokens_mask, overflowing]),
 Encoding(num_tokens=14, attributes=[ids, type_ids, tokens, offsets, attention_mask, special_tokens_mask, overflowing]),
 Encoding(num_tokens=90, attributes=[ids, type_ids, tokens, offsets, attention_mask, special_tokens_mask, overflowing]),
 Encoding(num_tokens=58, attributes=[ids, type_ids, tokens, offsets, attention_mask, special_tokens_mask, overflowing]),
 Encoding(num_tokens=1, attributes=[ids, type_ids, tokens, offsets, attention_mask, special_tokens_mask, overflowing])

What am I missing?