class_weight giving worst results in Keras model. What could be the reason?

user_11725056 0
Viewed 66

I'm working on an NLP Classification task with imbalanced data and the code:

df['target'] = le.fit_transform(df['CHAPTER'])
Y = df['target'].ravel()
classes = df['target'].nunique()

train_X, val_X, train_y, val_y = train_test_split(X,Y, test_size=0.1, stratify = Y, random_state = SEED)

class_weights = class_weight.compute_class_weight(class_weight = 'balanced',classes = np.unique(train_y),y = train_y)
class_weight_dict = dict(enumerate(class_weights))

vocab_size = 25000 

tokenizer = Tokenizer(num_words=vocab_size, filters = ' ')
tokenizer.fit_on_texts(list(train_X))
train_X = tokenizer.texts_to_sequences(train_X)
val_X = tokenizer.texts_to_sequences(val_X)

train_X = pad_sequences(train_X, maxlen=maxlen)
val_X = pad_sequences(val_X, maxlen=maxlen)

Works fine and giving me an accuracy of around 70% when I do:

history = model.fit(train_X, train_y, batch_size=64, epochs = 30,
                    validation_split = 0.1,verbose = 1)

But the moment I use class_weight=class_weight_dict in train , my accuracy drops from 70 to 30%. What could be the possible reason? Am I doing something wrong with the code?

scikit-learn keras tensorflow deep-learning machine-learning
1 Answers

When you use the dict(enumerate(class_weights)) method, it creates a dictionary with keys starting from zero. In case if you don't have class labels that correspond to zero (or if you don't have any in that range, at all) this can be a problem. Below is a demonstration:

train_y = [1, 1, 1, 2, 2]  # training labels: 1 and 2
class_weights = class_weight.compute_class_weight(
    class_weight='balanced',
    classes=np.unique(train_y),
    y=train_y
    )
print(class_weights)
# array([0.83333333, 1.25      ])

Creating the dictionary as you've done it:

class_weight_dict = dict(enumerate(class_weights))
print(class_weight_dict)
# {0: 0.8333333333333334, 1: 1.25}

There is no class as 0 and the class weight for class 2 is missing.

Instead, you should do:

class_weight_dict = {label: weight for label, weight in zip(np.unique(train_y), class_weights)}
print(class_weight_dict)
# {1: 0.8333333333333334, 2: 1.25}
user_58645820
Related