Giving less weight to data coming from another dataset that is noisy

user_19746111 0
Viewed 42

I have two datasets, one with clean data and one with dirty data. I train a Roberta model on the clean dataset and then get predictions for the dirty dataset. Those predictions with a probability greater than 0.9 go to the clean dataset. I then retrain the Roberta model with this new dataset (clean + dirty moving to clean).

For the retraining I am using the MAE loss function (more robust to noisy labels) and I use weights to give less value to the data that passes from the dirty to the clean dataset, as follows:

loss = torch.mean(torch.abs(y_true - y_pred) * weights)

Initially I am using an arbitrary weight of 0.5 for all the dirty data that gets passed into the clean dataset. However, I would like to assign them a weight in a more academic way, not so arbitrary.

How can I do that?

pytorch keras tensorflow deep-learning nlp
1 Answers

One way to choose the weight is based on your confidence in the dirty data and assign the weight accordingly. For example, if you think that 90% of dirty data is labeled correctly, then choosing 0.9 as the weight for the noisy data is a reasonable option.

Additionally, there is a whole literature on learning from noisy labels, you can check this survey for more information: https://arxiv.org/abs/2007.08199

user_129269220
Related