Using sample weights in SHAP explainers

user_8876307 0
Viewed 467

I want to calculate SHAP values for a linear model. For the regression, I have to use sample weights.

The problem is that I can´t evaluate if the sample weights were actually applied properly calculating the SHAP values.

Here is an example.

# Import libraries 

import shap
import pandas 
import numpy 
from sklearn.linear_model import LinearRegression
from sklearn.metrics import r2_score


# Setting up the data and the model

df.head()

weights Funnel  Q23_1   Q23_2   Q23_3   Q23_4   Q23_5   Q23_6   Q23_7    
Q23_8   Q23_9   Q23_10  Q23_11  Q23_12  Q23_13  Q23_14  Q23_15
847 0.75149 5.0 2.0 2.0 1.0 3.0 3.0 3.0 2.0 5.0 3.0 1.0 2.0 2.0 2.0 3.0 1.0
995 2.18378 2.0 1.0 1.0 1.0 3.0 3.0 3.0 1.0 4.0 2.0 2.0 1.0 2.0 2.0 2.0 2.0
14403   1.10852 2.0 1.0 1.0 1.0 2.0 2.0 4.0 1.0 5.0 1.0 2.0 2.0 1.0 3.0 3.0  
 1.0
13311   0.85934 4.0 2.0 2.0 3.0 3.0 2.0 3.0 3.0 4.0 4.0 3.0 2.0 2.0 3.0 3.0  
 2.0
17019   0.95337 2.0 1.0 1.0 2.0 3.0 2.0 2.0 3.0 2.0 2.0 2.0 3.0 1.0 1.0 1.0  
2.0

Y = df_t.drop(['Funnel', 'weights'], axis=1)
X = df_t[['Funnel']]

lm = LinearRegression()

First I calculate a regression without weights.

fit = lm.fit(X,Y)

pred = fit.predict(X)

print("R2 - No Weights:", r2_score(Y,pred))

Then I calculate a regression with weights.

fit = lm.fit(X,Y, sample_weight=df['weights'])

pred = fit.predict(X)

print("R2 - Wit weights:", r2_score(Y, pred2, sample_weight=df['weights']))

From was I figured out so far (I tested different combinations using other software packages e.g. R, SPSS to evaluate the results) is the I have to apply the weights to the fit() function and to r2_score() function to get the right result (see the example above). For example, if I only apply the weights to the fit() function but not to the r2_score() function the reported R2 value is wrong (i.e. the model is wrong). If I apply the weights also to the predict() function the R2 value is also wrong (i.e. the model is wrong).

fit = lm.fit(X,Y, sample_weight=df['weights'])

pred = fit.predict(X, sample_weight=df['weights'])

print("R2 - With something in between:", r2_score(Y, pred, sample_weight=df['weights']))

However, since I can calculate the SHAP values only in Python there is no way for me to evaluate the results. The question is how should I apply the sample weights to calculate the SHAP value right?

Only in the fit function (?):

fit = lm.fit(X,Y, sample_weight=df['weights'])
explainer = shap.LinearExplainer(fit, X, feature_dependence = 'independent')
shap_values = explainer.shap_values(X)

Or also in the explainer() function (?):

fit = lm.fit(X,Y, sample_weight=df['weights'])
explainer = shap.LinearExplainer(fit, X, feature_dependence = 'independent', 
sample_weight=df['weights'])
shap_values = explainer.shap_values(X)

There might be other possibilities... but I have no idea which one is right.

Here is a small data sample.

print(df.to_dict())
{'weights': {847: 0.75149, 995: 2.18378, 14403: 1.10852, 13311: 0.85934, 17019: 0.95337, 23707: 0.8899, 29562: 0.96819, 30627: 1.16261, 15187: 1.15915, 24179: 1.09833}, 'Funnel': {847: 5.0, 995: 2.0, 14403: 2.0, 13311: 4.0, 17019: 2.0, 23707: 2.0, 29562: 2.0, 30627: 4.0, 15187: 4.0, 24179: 5.0}, 'Q23_1': {847: 2.0, 995: 1.0, 14403: 1.0, 13311: 2.0, 17019: 1.0, 23707: 3.0, 29562: 1.0, 30627: 1.0, 15187: 5.0, 24179: 1.0}, 'Q23_2': {847: 2.0, 995: 1.0, 14403: 1.0, 13311: 2.0, 17019: 1.0, 23707: 2.0, 29562: 2.0, 30627: 1.0, 15187: 5.0, 24179: 1.0}, 'Q23_3': {847: 1.0, 995: 1.0, 14403: 1.0, 13311: 3.0, 17019: 2.0, 23707: 3.0, 29562: 2.0, 30627: 1.0, 15187: 5.0, 24179: 1.0}, 'Q23_4': {847: 3.0, 995: 3.0, 14403: 2.0, 13311: 3.0, 17019: 3.0, 23707: 3.0, 29562: 1.0, 30627: 1.0, 15187: 5.0, 24179: 1.0}, 'Q23_5': {847: 3.0, 995: 3.0, 14403: 2.0, 13311: 2.0, 17019: 2.0, 23707: 2.0, 29562: 3.0, 30627: 1.0, 15187: 5.0, 24179: 1.0}, 'Q23_6': {847: 3.0, 995: 3.0, 14403: 4.0, 13311: 3.0, 17019: 2.0, 23707: 4.0, 29562: 3.0, 30627: 1.0, 15187: 5.0, 24179: 2.0}, 'Q23_7': {847: 2.0, 995: 1.0, 14403: 1.0, 13311: 3.0, 17019: 3.0, 23707: 3.0, 29562: 2.0, 30627: 1.0, 15187: 5.0, 24179: 1.0}, 'Q23_8': {847: 5.0, 995: 4.0, 14403: 5.0, 13311: 4.0, 17019: 2.0, 23707: 4.0, 29562: 1.0, 30627: 1.0, 15187: 5.0, 24179: 1.0}, 'Q23_9': {847: 3.0, 995: 2.0, 14403: 1.0, 13311: 4.0, 17019: 2.0, 23707: 2.0, 29562: 1.0, 30627: 1.0, 15187: 5.0, 24179: 1.0}, 'Q23_10': {847: 1.0, 995: 2.0, 14403: 2.0, 13311: 3.0, 17019: 2.0, 23707: 2.0, 29562: 3.0, 30627: 1.0, 15187: 5.0, 24179: 1.0}, 'Q23_11': {847: 2.0, 995: 1.0, 14403: 2.0, 13311: 2.0, 17019: 3.0, 23707: 3.0, 29562: 2.0, 30627: 1.0, 15187: 2.0, 24179: 1.0}, 'Q23_12': {847: 2.0, 995: 2.0, 14403: 1.0, 13311: 2.0, 17019: 1.0, 23707: 2.0, 29562: 2.0, 30627: 1.0, 15187: 5.0, 24179: 1.0}, 'Q23_13': {847: 2.0, 995: 2.0, 14403: 3.0, 13311: 3.0, 17019: 1.0, 23707: 2.0, 29562: 4.0, 30627: 1.0, 15187: 5.0, 24179: 1.0}, 'Q23_14': {847: 3.0, 995: 2.0, 14403: 3.0, 13311: 3.0, 17019: 1.0, 23707: 3.0, 29562: 1.0, 30627: 1.0, 15187: 5.0, 24179: 1.0}, 'Q23_15': {847: 1.0, 995: 2.0, 14403: 1.0, 13311: 2.0, 17019: 2.0, 23707: 3.0, 29562: 1.0, 30627: 1.0, 15187: 5.0, 24179: 1.0}}
shap scikit-learn regression python
1 Answers

I face the same scenario. Here is my partial solution, I hope it is beneficial to you. Here my model is XGBoost instead of a linear learner.

  • I can insert the sample weight into the xgboost.DMatrix using the weight argument. And I put this matrix into explainer to get the shap values with reweighted samples. And I do the absolute value calculation to get the bar plot for feature importances.
  • Alternatively, try the pandas.DataFrame.sample function to do the resampling using the weight argument. Next, do the similar procedure like what you do using shap package.

Both solutions works for me.

user_86252280
Related