I have 6 different classes which I am doing multiclass classification on using both XGBoost and RandomForest.
What I want is to analyze which features are most important for a sample belonging to each class.
I know that there are two different ways of getting the feature importance using xgboost. First the built in feature_importances_ variable which returns an array with importance score for all features.
Another way is to from xgboost import plot_importance which you can use to provide a plot through
plot_importance(model = xgb, max_num_features = 20)
But all of these methods only return the most important features without respect to classes.
Is there a way to get the most important features for a sample belonging to each class?
Or would the solution be to create 6 different binary classifiers and try to analyze those feature importances instead?