Is it okay to build a model on imbalanced data?

user_8553795 0
Viewed 578

Background - The dataset I am working on is highly imbalanced and the number of classes is 543. The data is bounded by date. After exploring the data over a span of 5 years I came to know the imbalance is inherent and its persistent. The test data which the model will get will also be bounded by a date range and it will also have a similar imbalance.

The reason for the imbalance in the data is different amount of spend, popularity of a product. Handling imbalance would do injustice to the business.

Questions - In such a case, is it okay to proceed with building model on imbalanced data?

The model would be retrained every month on the new data and it would be used for predictions once in a month.

imbalanced-data scikit-learn machine-learning
1 Answers

Depending on what you are trying to model, it may or may not be correct to do so.

Training on an imbalanced dataset will generally make your model overfit those elements that appear more often, which leads to bias towards those ones at best or no understanding of the underrepresented samples at worst. If you are trying to model the natural occurrences of some information, then an unbalanced dataset in essence has a prior probability applied to it already, so the resulting bias may be desired. In these cases, the number of elements per class, say, is part of the actual information. Such a bias can be (un-)modeled artificially too, however, e.g. by applying a scaling factor for classification (e.g. through class weights), etc. To avoid such bias, boosting and ensemble methods such as Xgboost (or Adaboost in more trivial cases) or just Random Forests work relatively well. If you have the time, k-fold cross validation can help reducing the error further.

To make sure every sample is adequately represented, you may choose to oversample the underrepresented classes or undersample the overrepresented ones. In order to determine correct likelihoods, make sure to capture the prior distribution as well and use it to shape the posterior. Data augmentation may help you if the number of samples is low; depending on your case, synthetic data generation might be a good approach. You could try, say, training a GAN only on the underrepresented samples and use that to generate more - as in idea: train it on all available data first, then change the discriminator loss to force it to forge and recognize the underrepresented classes only. Without entering the Deep Learning domain, techniques such as SMOTE or ADASYN may work. Both are available in the imblearn Python package which builds on scikit-learn.

Lastly, carefully selecting the loss metric may help. You can find more (and more detailed) information in papers such as Survey on deep learning with class imbalance.

user_1956510
Related