What do the logits and probabilities from RobertaForSequenceClassification represent?

user_4847912 0
Viewed 4226

Being new to the "Natural Language Processing" scene, I am experimentally learning and have implemented the following segment of code:

from transformers import RobertaTokenizer, RobertaForSequenceClassification
import torch
    
path = "D:/LM/rb/"
tokenizer = RobertaTokenizer.from_pretrained(path)
model = RobertaForSequenceClassification.from_pretrained(path)
    
inputs = tokenizer("Hello, my dog is cute", return_tensors="pt")
outputs = model(**inputs)
pred_logits = outputs.logits
print(pred_logits)
probs = pred_logits.softmax(dim=-1).detach().cpu().flatten().numpy().tolist()
print(probs)

I understand that applying the model returns a "torch.FloatTensor comprising various elements depending on the configuration (RobertaConfig) and inputs", and that the logits are accessible using .logits. As demonstrated I have applied the .softmax function to the tensor to return normalised probabilities and have converted the result into a list. I am outputted with the following:

[0.5022980570793152, 0.49770188331604004]

Do these probabilities represent some kind of overall "masked" probability?

What do the first and second index represent in context of the input?

EDIT:

model.num_labels

Output:

2

@cronoik explains that the model "tries to classify if a sequence belongs to one class or another"

Am I to assume that because there are no trained output layers these classes don't mean anything yet?

For example, I can assume that the probability that the sentence, post analysis, belongs to class 1 is 0.5. However, what is class 1?

Additionally, model cards with pre-trained output layers such as the open-ai detector help differentiate between what is "real" and "fake", and so I can assume the class that a sentence belongs to. However, how can I confirm these "labels" without some type of "mapping.txt" file?

text-classification huggingface-transformers pytorch nlp python
1 Answers

You have initialized a RobertaForSequenceClassification model that per default (in case of roberta-base and roberta-large which have no trained output layers for sequence classification) tries to classify if a sequence belongs to one class or another. I used the expression "belongs to one class or another" because these classes have no meaning yet. The output layer is untrained and it requires a finetuning to give these classes a meaning. Class 0 could be X and Class 1 could be Y or the other way around. For example, the tutorial for finetuning a sequence classification model for the IMDb review dataset defines negative reviews as Class 0 and positive reviews as Class 1 (link).

You can check the number of supported classes with:

model.num_labels

Output:

2

The output you get is the non-normalized probability for each class (i.e. logits). You applied the softmax function to normalize these probabilities, which leads to 0.5022980570793152 for the first class and 0.49770188331604004 for the second class.

Maybe you got confused because the values are close to each other. Let's try a model with a pretrained output layer (model card):

sentimodel = RobertaForSequenceClassification.from_pretrained('cardiffnlp/twitter-roberta-base-sentiment')
print(sentimodel.num_labels)
outputs = sentimodel(**inputs)
print(outputs.logits.softmax(dim=-1).tolist())

Output:

3
[[0.0015561950858682394, 0.019568447023630142, 0.9788752794265747]]

These values represent the probabilities for the sentence Hello, my dog is cute to be negative, neutral, or positive. We know what these classes are because the authors provided mapping that clarifies it. In case the authors of the model do not provide such a mapping (via a readme or the original training code), we can only guess what each class represents by testing it with random samples.

The model card you have mentioned does not provide any useful information regarding the mapping of the classes to what they represent, but the model is provided by huggingface itself and they provide a link to the code used for training the model. The dataset.py indicates that fake is represented by Class 0 and real by Class 1.

user_66648720
Related