How to get variance explained of features for lightgbm model?

user_11846355 0
Viewed 993

I used lightgbm for feature importance. However, the output is a plot scores by some metric. My questions are:

  1. What are is the metric in the x-axis? Is that an F-score or something else?
  2. How can I get an output of the features where it shows me how much each feature makes up for the variance the model (similar to PCA)?
  3. How do I extract the Metric for all the feature of importance in a dataframe format?

This is my code:

import lightgbm as lgb
import matplotlib.pyplot as plt

lgb_params = {
    'boosting_type': 'gbdt',
    'objective': 'binary',
    'num_leaves': 30,
    'num_round': 360,
    'max_depth':8,
    'learning_rate': 0.01,
    'feature_fraction': 0.5,
    'bagging_fraction': 0.8,
    'bagging_freq': 12
}

lgb_train = lgb.Dataset(X, y)
model = lgb.train(lgb_params, lgb_train)

plt.figure(figsize=(12,6))
lgb.plot_importance(model, max_num_features=30)
plt.title("Feature importances")
plt.show()

enter image description here

feature-selection lightgbm hyperparameters machine-learning python
1 Answers

1) the metric on x axis, in your case, is the feature importance obtained with "split" type (by default). as you can see in lgm doc: the importance can be calculated using "split" or "gain" method. If "split", result contains numbers of times the feature is used in a model. If "gain", result contains total gains of splits that use the feature.

The first measure is split-based, it doesn’t take the number of samples into account. The second measure is gain-based. It’s basically the same as the method in scikit-learn with Gini impurity replaced by the objective used by the gradient boosting model

These measures are purely calculated using training data, so there’s a chance that a split creates no improvement on the objective in test-set

2) the most similar measure to explained_variance_ratio_ of sklearn pca (not in the meaning but in the way it can be used) is exactly feature_importances in tree-based method. if you are more confident you can scale this number in % as done by sklearn random forest, where the feature importances sum up to 1. you can do model.feature_importances_/model.feature_importances_.sum(). Otherwise, there are other similar methods like permutation importance

3) to store in df all the importances you can do: pd.DataFrame({'name':model.feature_name_,'importance':model.feature_importances_})

user_103750490
Related