Model overfits after first epoch

Question

I'm trying to use hugging face's BERT-base-uncased model to train on emoji prediction on tweets, and it seems that after the first epoch, the model immediately starts to overfit. I have tried the following:

Increasing the training data (I increased this from 1x to 10x with no effect)
Changing the learning rate (no differences there)
Using different models from hugging face (the results were the same again)
Changing the batch size (went from 32, 72, 128, 256, 512, 1024)
Creating a model from scratch, but I ran into issues and decided to post here first to see if I was missing anything obvious.

At this point, I'm concerned that the individual tweets don't give enough information for the model to make a good guess, but wouldn't it be random in that case, rather than overfitting?

Also, training time seems to be ~4.5 hours on Colab's free GPUs, is there any way to speed that up? I tried their TPU, but it doesn't seem to be recognized.

This is what the data looks like

And this is my code below:

import pandas as pd
import json
import re
from transformers import AutoTokenizer, AutoModelForSequenceClassification
from sklearn.model_selection import train_test_split
import torch
from transformers import TrainingArguments, Trainer
from transformers import EarlyStoppingCallback
from sklearn.metrics import accuracy_score,precision_score, recall_score, f1_score
import numpy as np

# opening up the data and removing all symbols
df = pd.read_json('/content/drive/MyDrive/computed_results.json.bz2')
df['text_no_emoji'] = df['text_no_emoji'].apply(lambda text: re.sub(r'[^\w\s]', '', text))


# loading the tokenizer and the model from huggingface
tokenizer = AutoTokenizer.from_pretrained("bert-base-uncased")
model = AutoModelForSequenceClassification.from_pretrained("bert-base-uncased", num_labels=5).to('cuda')

# test train split
train, test = train_test_split(df[['text_no_emoji', 'emoji_codes']].sample(frac=1), test_size=0.2)

# defining a dataset class that generates the encoder and labels on the fly to minimize memory usage
class Dataset(torch.utils.data.Dataset):    
    def __init__(self, input, labels=None):
        self.input = input
        self.labels = labels

    def __getitem__(self, pos):
        encoded = tokenizer(self.input[pos], truncation=True, max_length=15, padding='max_length')
        label = self.labels[pos]
        ret = {key: torch.tensor(val) for key, val in encoded.items()}

        ret['labels'] = torch.tensor(label)
        return ret

    def __len__(self):
        return len(self.labels)

# training and validation datasets are defined here
train_dataset = Dataset(train['text_no_emoji'].tolist(), train['emoji_codes'].tolist())
val_dataset = Dataset(train['text_no_emoji'].tolist(), test['emoji_codes'].tolist())

# defining the training arguments
args = TrainingArguments(
    output_dir="output",
    evaluation_strategy="epoch",
    logging_steps = 10,
    per_device_train_batch_size=1024,
    per_device_eval_batch_size=1024,
    num_train_epochs=5,
    save_steps=3000,
    seed=0,
    load_best_model_at_end=True,
    weight_decay=0.2,
)

# defining the model trainer
trainer = Trainer(
    model=model,
    args=args,
    train_dataset=train_dataset,
    eval_dataset=val_dataset
)

# Training the model
trainer.train()

Results: After this, the training generally stops pretty fast due to the early stopper

The dataset can be found here (39 Mb compressed)

Model overfits after first epoch

0 Answers