SHAP values with PyTorch - KernelExplainer vs DeepExplainer

user_439181 0
Viewed 2302

I haven't been able to find much in the way of examples on SHAP values with PyTorch. I've used two techniques to generate SHAP values, however, their results don't appear to agree with each other.

SHAP KernelExplainer with PyTorch

import torch
from torch.autograd import Variable
import shap
import numpy
import pandas

torch.set_grad_enabled(False)

# Get features
train_features_df = ... # pandas dataframe
test_features_df = ... # pandas dataframe

# Define function to wrap model to transform data to tensor
f = lambda x: model_list[0]( Variable( torch.from_numpy(x) ) ).detach().numpy()

# Convert my pandas dataframe to numpy
data = test_features_df.to_numpy(dtype=np.float32)

# The explainer doesn't like tensors, hence the f function
explainer = shap.KernelExplainer(f, data)

# Get the shap values from my test data
shap_values = explainer.shap_values(data)

# Enable the plots in jupyter
shap.initjs()

feature_names = test_features_df.columns
# Plots
#shap.force_plot(explainer.expected_value, shap_values[0], feature_names)
#shap.dependence_plot("b1_price_avg", shap_values[0], data, feature_names)
shap.summary_plot(shap_values[0], data, feature_names)

SHAP summary plot from KernelExplainer with PyTorch

SHAP DeepExplainer with PyTorch

# It wants gradients enabled, and uses the training set
torch.set_grad_enabled(True)
e = shap.DeepExplainer(model, Variable( torch.from_numpy( train_features_df.to_numpy(dtype=np.float32) ) ) )

# Get the shap values from my test data (this explainer likes tensors)
shap_values = e.shap_values( Variable( torch.from_numpy(data) ) )

# Plots
#shap.force_plot(explainer.expected_value, shap_values, feature_names)
#shap.dependence_plot("b1_price_avg", shap_values, data, feature_names)
shap.summary_plot(shap_values, data, feature_names)

enter image description here

Comparing results

As you can see from the summary plots, the value given to the features from the same PyTorch model, with the same test data, are noticeably different.

For example the feature b1_addresses_avg has value one from last with the KernelExplainer. But with the DeepExplainer is ranked third from top.

I'm not sure where to go from here.

shap pytorch python
1 Answers

Shapley values are very difficult to calculate exactly. Kernel SHAP and Deep SHAP are two different approximation methods to calculate the Shapley values efficiently, and so one shouldn't expect them to necessarily agree.

You can read the authors' paper for more details.

While Kernel SHAP can be used on any model, including deep models, it is natural to ask whether there is a way to leverage extra knowledge about the compositional nature of deep networks to improve computational performance. [...] This motivates our adapting DeepLIFT to become a compositional approximation of SHAP values, leading to Deep SHAP.

In section 5, they compare the performance of Kernel SHAP and Deep SHAP. From their example it seems like Kernel SHAP performs better than Deep SHAP. So I guess if you aren't running into computational issues, you can stick with Kernel SHAP.

Figure 5B

p.s. Just to make sure, you're inputting the exact same trained model to SHAP right? You shouldn't be training separate models, because they'll learn different weights.

user_109883470
Related