Understanding pos_weight argument in BCEWithLogitsLoss

user_3687776 0
Viewed 1553

I am trying to understand how the pos_weight argument is being used in BCEWithLogitsLoss in order to be able to correctly define the pos_weight Tensor. The documentation only mentions: "a weight of positive examples. Must be a vector with length equal to the number of classes.". As I was not able to get a sufficient understanding by looking at the code (the actual code is hidden behind multiple function loaders and I didn't even manage to reach a point where pos_weight is used), I have several questions regarding the pos_weight argument:

  1. Is the weight for negative samples always 1?
  2. If the weight for negative samples is always 1 and assuming I want every sample to contribute equally to the loss I would do something like the following. Let l = [100, 10, 5, 15] where l[0] is the amount of negative samples and l[1:] is the amount of positive samples from each label. In pseudo code this looks like:
l = [100, 10, 5, 15]
lcm = LCM(l)  # 300
weights = lcm / l  # weights = [3, 30, 60, 20]
weights = weights / l[0]  # weights = [1, 10, 20, 6.6667]
positive_weights = weights[1:]  # [10, 20, 6.66667]
criterion = nn.BCEWithLogitsLoss(pos_weight=positive_weights)

Can someone please confirm if my understanding of how pos_weight should be used is correct?

pytorch
1 Answers

I didn't manage to find a definitive answer but from my experiments it seems I was right, i.e. negative sample weights are considered to be 1 and increasing the weights past 1 for positive samples makes them more important. Additionally, the LCM approach for figuring out the weights is actually both not good and not needed. First it is bad because if you have a lot of classes with different number of samples the LCM can be an incredibly large number so very often computing it exactly is not even feasible. Additionally, it is not needed because you achieve the same weights by simply dividing the amount of negative samples with the amount of positive samples for every label and obtain the weights in that manner. Here is a short code snippet on how you can obtain the weights if you have your data in pandas DataFrame. Obtaining the weights in this way results in a 'balanced' dataset where ideally you optimize for both precision and recall.

def get_train_dataset_pos_weights(self):
    """Returns weights for positive samples used for unbalanced datasets."""

    num_pos_samples = self.train_df[self.label_columns].sum().sum()
    num_neg_samples = len(self.train_df) - num_pos_samples
    assert len(self.train_df) == (num_neg_samples + num_pos_samples)
    num_pos_samples_per_label = np.array(
        list(self.train_df[self.label_columns].sum()))
    weights = num_neg_samples / num_pos_samples_per_label

    return torch.from_numpy(weights)
user_36877760
Related