Using shap library for model understanding when model input is a list of tensors

user_7009988 0
Viewed 103

I'm fairly new to deep learning models, how to design them, and how to understand them. I'm trying to understand my model using the SHAP library, but having some issues with following this tutorial.

My model's forward function looks like this, where the input is a list of tensors.

def forward(self, batch, runValidation=False):
        [
            visual_features, token_ids, token_masks, labels, xpath_ids,
            class_ids, attr_token_masks
        ] = batch
        ...
        return loss, class_loss, pr_result, output, visual_features

But based on the tutorial and error messages I'm getting, it seems that SHAP expects the model inputs and outputs to be tensors. Namely, background and x_test_each_class seem to be tensors.

# select backgroud for shap
background = x_train[np.random.choice(x_train.shape[0], 1000, replace=False)]

# DeepExplainer to explain predictions of the model
explainer = shap.DeepExplainer(model, background)

# compute shap values
shap_values = explainer.shap_values(x_test_each_class)

My questions are:

  1. Is this proper model design to have the input/output be a list of tensors?
  2. If yes, then is it still possible for me to use SHAP on this model?
shap deep-learning python
1 Answers

I ended up concatenating my list of inputs into one long tensor, then writing a wrapper around my forward function to parse those inputs back into their original format before feeding them into the forward function.

user_70099880
Related