Getting Invalid argument error for Binary classification

Question

I am working on the DistillBert project for binary classification. I am trying to run the following code using the Spam SMS data set (You can also use the IMDB dataset, it is also giving the same issue), I am trying to find out the recall, precision, and AUC score. However, I am getting an Invalid Argument error.

Also, the prediction is like [-9., 9.] . Why is there a negative value here when it is the probability of the class?

Here I am using the BinaryCrossentropy loss function and Adam optimizer.

Dataset: The dataset used here is the spam SMS dataset which has binary labels 0 for normal SMS and 1 for spam SMS. The same error can be reproduced using the IMDB data set for this code.

Error:

InvalidArgumentError: predictions must be >= 0
Condition x >= y did not hold.
First 3 elements of x:
[-9.  9. -9.]
First 1 elements of y:
[0.]

Code:

import pandas as pd
import tensorflow as tf
import transformers
from transformers import DistilBertTokenizer
from transformers import TFAutoModelForSequenceClassification
pd.set_option('display.max_colwidth', None)
MODEL_NAME = 'distilbert-base-uncased'
BATCH_SIZE = 8
N_EPOCHS = 3

train = pd.read_csv("train_set.csv", error_bad_lines=False)
test = pd.read_csv("test_set.csv", error_bad_lines=False)

X_train = train.text
X_test = test.text
y_train = train.label
y_test = test.label

#One-hot encoding of labels
y_train_encoded = tf.one_hot(y_train.values, 2)
y_test_encoded = tf.one_hot(y_test.values, 2)

tokenizer = DistilBertTokenizer.from_pretrained(MODEL_NAME)

train_encodings = tokenizer(list(X_train.values),
                        truncation=True, 
                        padding=True)
test_encodings = tokenizer(list(X_test.values),
                       truncation=True, 
                       padding=True)

train_dataset = 
tf.data.Dataset.from_tensor_slices((dict(train_encodings),list(y_train_encoded)))

test_dataset = 
tf.data.Dataset.from_tensor_slices((dict(test_encodings),list(y_test_encoded)))
test_dataset2 = test_dataset.shuffle(buffer_size=1024).take(1000).batch(16)

model = TFAutoModelForSequenceClassification.from_pretrained(MODEL_NAME)

optimizerr = tf.keras.optimizers.Adam(learning_rate=5e-5)

losss = tf.keras.losses.BinaryCrossentropy((from_logits=True)

model.compile(optimizer=optimizerr,
          loss=losss,
          metrics=['accuracy'])

print("Evaluate Base model on test data")
results = model.evaluate(test_dataset2)
print("test loss, test acc:", results)

model.fit(train_dataset.shuffle(len(X_train)).batch(BATCH_SIZE),
      epochs=N_EPOCHS,
      batch_size=BATCH_SIZE)

predictions = model.predict(test_dataset2)
# predictions = TFSequenceClassifierOutput(loss=None, logits=array([[-8.96754 ,  8.966875],....[-8.970767,  8.975923]], dtype=float32), hidden_states=None, attentions=None)

import numpy as np
rounded_predictions = np.rint(predictions.logits)
# rounded_predictions = array([[-9.,  9.],....[-9.,  9.]], dtype=float32)

y_test_encoded
#y_test_encoded = 

m = tf.keras.metrics.Recall()
m.update_state(y_test_encoded, rounded_predictions)
m.result().numpy()
# This is where I getting the above mentioned error.

How can I fix the error and get the recall, precision and AUC scores?

Getting Invalid argument error for Binary classification

0 Answers