Where in the code of pytorch or huggingface/transformer label gets "renamed" into labels?

user_5944744 0
Viewed 1422

My question concerns the example, available in the great huggingface/transformers library.

I am using a notebook, provided by library creators as a starting point for my pipeline. It presents a pipeline of finetuning a BERT for Sentence Classification on Glue dataset.

When getting into the code, I noticed a very weird thing, which I cannot explain.

In the example, input data is introduced to the model as the instances of the InputFeatures class from here:

This class has 4 attributes, including the label attribute:

class InputFeatures:
    ...
    input_ids: List[int]
    attention_mask: Optional[List[int]] = None
    token_type_ids: Optional[List[int]] = None
    label: Optional[Union[int, float]] = None

which are later passed as a dictionary of inputs to the forward() method of the model. This is done by the Trainer class, for example in the lines 573-576 here:

    def _training_step(
        self, model: nn.Module, inputs: Dict[str, torch.Tensor], optimizer: torch.optim.Optimizer
    ) -> float:
        model.train()
        for k, v in inputs.items():
            inputs[k] = v.to(self.args.device)

        outputs = model(**inputs)

However, the forward() method expects labels (note the plural form) input parameter (taken from here):

    def forward(
        self,
        input_ids=None,
        attention_mask=None,
        head_mask=None,
        inputs_embeds=None,
        labels=None,
        output_attentions=None,
    ):

So my question is where does the label become labels in this pipeline?

To give some extra info on the issue, I created my own pipeline, which uses nothing, related, with Glue data and pipe, basically it relies only on the Trainer class of transformers. I even use another model (Flaubert). I replicated the InputFeature class and my code works for both cases below:

class InputFeature:
    def __init__(self, text, label):
        self.input_ids = text
        self.label = label

class InputFeaturePlural:
    def __init__(self, text, label):
        self.input_ids = text
        self.labels = label

But it does not work if I name the second attribute as self.labe or by any other names. Why is it possible to use both attribute names?

It's not like it is extremely important in my case, but I feel uncomfortable passing around the data in the variable, which "changes name" somewhere along the way.

huggingface-transformers pytorch python
1 Answers

The rename happens in the collator. In the trainer init, when data_collator is None, a default one is used:

class Trainer:
    # ...
    def __init__(...):
        # ...
        self.data_collator = data_collator if data_collator is not None else default_data_collator
        # ...

FYI, the self.data_collator is later used when you get the dataloader:

data_loader = DataLoader(
    self.train_dataset,
    batch_size=self.args.train_batch_size,
    sampler=train_sampler,
    collate_fn=self.data_collator,              # <-- here
    drop_last=self.args.dataloader_drop_last,
)

The default collator has a special handling for labels, which does this renaming, if needed:

# Special handling for labels.
# Ensure that tensor is created with the correct type
# (it should be automatically the case, but let's make sure of it.)
if hasattr(first, "label") and first.label is not None:
    if type(first.label) is int:
        labels = torch.tensor([f.label for f in features], dtype=torch.long)
    else:
        labels = torch.tensor([f.label for f in features], dtype=torch.float)
    batch = {"labels": labels}  # <-- here is where it happens
elif hasattr(first, "label_ids") and first.label_ids is not None:
    if type(first.label_ids[0]) is int:
        labels = torch.tensor([f.label_ids for f in features], dtype=torch.long)
    else:
        labels = torch.tensor([f.label_ids for f in features], dtype=torch.float)
    batch = {"labels": labels}
else:
    batch = {}
user_42282750
Related