I am working on the DistillBert project for binary classification. I am trying to run the following code using the Spam SMS data set (You can also use the IMDB dataset, it is also giving the same issue), I am trying to find out the recall, precision, and AUC score. However, I am getting a Value error. The error is occurring at the end of the code given below i.e. after this code block.
m = tf.keras.metrics.Recall()
m.update_state(y_test_encoded, rounded_predictions)
m.result().numpy()
The error can be generated using the code below and the SMS spam data split into train and test set which can be found here - SMS spam dataset
Here I am using the BinaryCrossentropy loss function and Adam optimizer.
Dataset: The dataset used here is the spam SMS dataset which has binary labels 0 for standard SMS and 1 for spam SMS. The same error can be reproduced using the IMDB data set for this code.
Error traceback:
---------------------------------------------------------------------------
ValueError Traceback (most recent call last)
/scratch/local/46806264/ipykernel_20971/1328487267.py in <module>
1 m = tf.keras.metrics.Recall()
----> 2 m.update_state(y_test_encoded, y_pred)
3 m.result().numpy()
/apps/tensorflow/2.4.1.cuda11/lib/python3.8/site-packages/tensorflow/python/keras/utils/metrics_utils.py in decorated(metric_obj, *args, **kwargs)
88
89 with tf_utils.graph_context_for_symbolic_tensors(*args, **kwargs):
---> 90 update_op = update_state_fn(*args, **kwargs)
91 if update_op is not None: # update_op will be None in eager execution.
92 metric_obj.add_update(update_op)
/apps/tensorflow/2.4.1.cuda11/lib/python3.8/site-packages/tensorflow/python/keras/metrics.py in update_state_fn(*args, **kwargs)
175 control_status = ag_ctx.control_status_ctx()
176 ag_update_state = autograph.tf_convert(obj_update_state, control_status)
--> 177 return ag_update_state(*args, **kwargs)
178 else:
179 if isinstance(obj.update_state, def_function.Function):
/apps/tensorflow/2.4.1.cuda11/lib/python3.8/site-packages/tensorflow/python/autograph/impl/api.py in wrapper(*args, **kwargs)
665 try:
666 with conversion_ctx:
--> 667 return converted_call(f, args, kwargs, options=options)
668 except Exception as e: # pylint:disable=broad-except
669 if hasattr(e, 'ag_error_metadata'):
/apps/tensorflow/2.4.1.cuda11/lib/python3.8/site-packages/tensorflow/python/autograph/impl/api.py in converted_call(f, args, kwargs, caller_fn_scope, options)
348 if conversion.is_in_allowlist_cache(f, options):
349 logging.log(2, 'Allowlisted %s: from cache', f)
--> 350 return _call_unconverted(f, args, kwargs, options, False)
351
352 if ag_ctx.control_status_ctx().status == ag_ctx.Status.DISABLED:
/apps/tensorflow/2.4.1.cuda11/lib/python3.8/site-packages/tensorflow/python/autograph/impl/api.py in _call_unconverted(f, args, kwargs, options, update_cache)
476
477 if kwargs is not None:
--> 478 return f(*args, **kwargs)
479 return f(*args)
480
/apps/tensorflow/2.4.1.cuda11/lib/python3.8/site-packages/tensorflow/python/keras/metrics.py in update_state(self, y_true, y_pred, sample_weight)
1404 Update op.
1405 """
-> 1406 return metrics_utils.update_confusion_matrix_variables(
1407 {
1408 metrics_utils.ConfusionMatrix.TRUE_POSITIVES: self.true_positives,
/apps/tensorflow/2.4.1.cuda11/lib/python3.8/site-packages/tensorflow/python/keras/utils/metrics_utils.py in update_confusion_matrix_variables(variables_to_update, y_true, y_pred, thresholds, top_k, class_id, sample_weight, multi_label, label_weights)
352 losses_utils.squeeze_or_expand_dimensions(
353 y_pred, y_true, sample_weight=sample_weight))
--> 354 y_pred.shape.assert_is_compatible_with(y_true.shape)
355
356 if top_k is not None:
/apps/tensorflow/2.4.1.cuda11/lib/python3.8/site-packages/tensorflow/python/framework/tensor_shape.py in assert_is_compatible_with(self, other)
1132 """
1133 if not self.is_compatible_with(other):
-> 1134 raise ValueError("Shapes %s and %s are incompatible" % (self, other))
1135
1136 def most_specific_compatible_shape(self, other):
ValueError: Shapes (426530, 2) and (1930, 2) are incompatible
The shape of the ground truth label is (1930,2) and that of the predicted label is (426530,2). This is because the shape of the input_ids and attention_mask of one element in the test_dataset is (221,). 1930x221=426530. I have given the values and shapes of the test_dataset below. I am using test_dataset for prediction. I don't understand how is this happening. How is the model making predictions for all the elements in the tokenized dataset? How can I fix this to get my model's matrix?
Value of test_dataset:test_dataset
<TensorSliceDataset shapes: ({input_ids: (221,), attention_mask: (221,)}, (2,)), types: ({input_ids: tf.int32, attention_mask: tf.int32}, tf.float32)>
1st element of test_dataset: On running command-
for x, y in test_dataset.take(1):
print(x)
output-
array([ 101, 2272, 2000, 14163, 1010, 2057, 1005, 2128, 22210,
2041, 2256, 27290, 3663, 102, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0], dtype=int32)>, 'attention_mask': <tf.Tensor: shape=(221,), dtype=int32, numpy=
array([1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0], dtype=int32)>}
Code:
import pandas as pd
import tensorflow as tf
import transformers
from transformers import DistilBertTokenizer
from transformers import TFAutoModelForSequenceClassification
pd.set_option('display.max_colwidth', None)
MODEL_NAME = 'distilbert-base-uncased'
BATCH_SIZE = 8
N_EPOCHS = 3
train = pd.read_csv("train_set.csv", error_bad_lines=False)
test = pd.read_csv("test_set.csv", error_bad_lines=False)
X_train = train.text
X_test = test.text
y_train = train.label
y_test = test.label
#One-hot encoding of labels
y_train_encoded = tf.one_hot(y_train.values, 2)
y_test_encoded = tf.one_hot(y_test.values, 2)
tokenizer = DistilBertTokenizer.from_pretrained(MODEL_NAME)
train_encodings = tokenizer(list(X_train.values),
truncation=True,
padding=True)
test_encodings = tokenizer(list(X_test.values),
truncation=True,
padding=True)
train_dataset =
tf.data.Dataset.from_tensor_slices((dict(train_encodings),list(y_train_encoded)))
test_dataset =
tf.data.Dataset.from_tensor_slices((dict(test_encodings),list(y_test_encoded)))
test_dataset2 = test_dataset.shuffle(buffer_size=1024).take(1000).batch(16)
model = TFAutoModelForSequenceClassification.from_pretrained(MODEL_NAME)
optimizerr = tf.keras.optimizers.Adam(learning_rate=5e-5)
losss = tf.keras.losses.BinaryCrossentropy((from_logits=True)
model.compile(optimizer=optimizerr,
loss=losss,
metrics=['accuracy'])
print("Evaluate Base model on test data")
results = model.evaluate(test_dataset2)
print("test loss, test acc:", results)
model.fit(train_dataset.shuffle(len(X_train)).batch(BATCH_SIZE),
epochs=N_EPOCHS,
batch_size=BATCH_SIZE)
predictions = model.predict(test_dataset)
y_pred = tf.round(tf.nn.sigmoid(predictions.logits))
m = tf.keras.metrics.Recall()
m.update_state(y_test_encoded, rounded_predictions)
m.result().numpy()
#the above-mentioned error is occurring here.
How can I fix the error and get the recall, precision and AUC scores?