How to set class weights in DecisionTreeClassifier for multi-class setting

user_9907733 0
Viewed 5061

I am using sklearn.tree.DecisionTreeClassifier to train 3-class classification problem.

The number of records in 3 classes are given below:

A: 122038
B: 43626
C: 6678

When I train the classifier model it fails to learn the class - C. Though efficiency comes out to be 65-70% but it completely ignores the class C.

Then I came to know about class_weight parameter but I am not sure how to use it in multiclass setting.

Here is my code: ( I used balanced but it gave more poor accuracy)

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=1)
clf = tree.DecisionTreeClassifier(criterion="gini", max_depth=3, random_state=1,class_weight='balanced')
clf = clf.fit(X_train,y_train)
y_pred = clf.predict(X_test)

How can I use weights with proportion to class distributions.

Secondly, is there any better way to address this Imbalance class problem to increase accuracy.?

scikit-learn decision-tree machine-learning python
2 Answers

You can also pass a dictionary of values to the class_weight argument in order to set your own weights. For example to weight class A half as much you could do:

class_weight={
    'A': 0.5,
    'B': 1.0,
    'C': 1.0
}

By doing class_weight='balanced' it automatically sets the weights inversely proportional to class frequencies.

More information can be found in the docs under the class_weight argument: https://scikit-learn.org/stable/modules/generated/sklearn.tree.DecisionTreeClassifier.html

It is usually to be expected that balancing the classes would reduce accuracy. This is why accuracy is often considered to be a poor metric for imbalanced data sets.

You can try the Balanced Accuracy metric that sklearn includes to start, but there are many other potential metrics to try which will depend on what your ultimate goal is.

https://scikit-learn.org/stable/modules/model_evaluation.html

If you are not familiar with the 'confusion matrix' and its related values like precision and recall then I would start your research there.

https://en.wikipedia.org/wiki/Precision_and_recall

https://en.wikipedia.org/wiki/Confusion_matrix

https://scikit-learn.org/stable/auto_examples/model_selection/plot_confusion_matrix.html

user_100063760

The “balanced” mode is the way to start.

The “balanced” mode uses the values of y to automatically adjust weights inversely proportional to class frequencies in the input data as n_samples / (n_classes * np.bincount(y))

To manually define the weights, you need a dictionary or a list of dictionaries depending on the problem.

class_weight dict, list of dict or “balanced”, default=None

Weights associated with classes in the form {class_label: weight}. If None, all classes are supposed to have weight one. For multi-output problems, a list of dicts can be provided in the same order as the columns of y.

Note that for multioutput (including multilabel) weights should be defined for > each class of every column in its own dict. For example, for four-class multilabel > classification weights should be [{0: 1, 1: 1}, {0: 1, 1: 5}, {0: 1, 1: 1}, {0: 1, 1: 1}] instead of [{1:1}, {2:5}, {3:1}, {4:1}].

Example:

If the frequency of class A is 10% and the frequency of class B is 90%:

clf = tree.DecisionTreeClassifier(class_weight={A:9,B:1})
user_50250090
Related