I'm training a classification model, and I've decided to switch from categorical crossentropy loss function to sparse categorical crossentropy to potentially use less memory and have faster trainings. My training computes precision and recall metrics.
However, when I switch to sparse crossentropy, precision metric starts to fail. The thing is that SparseCategoricalCrossentropy expects true labels to be scalars, while predicted labels to be vectors of size "number of classes", and precision metrics raises an exception of "shape mistmatch" type.
A minimal example to show this (the same model works without the precision score, and fails during the second training with added precision score computation):
import numpy as np
import tensorflow as tf
x = np.arange(0, 20)
y = np.zeros_like(x)
for i in range(len(x)):
if x[i] % 2 == 0:
y[i] = 0 # Even number
else:
y[i] = 1 # Odd number
n_classes = len(np.unique(y))
model = tf.keras.Sequential(
[
tf.keras.layers.Dense(10, input_shape=(1,)),
tf.keras.layers.Dense(n_classes, activation="softmax"),
]
)
print("Train without precision metric")
model.compile(
optimizer="adam",
loss="sparse_categorical_crossentropy",
)
model.fit(x, y, epochs=2)
print("Train with precision metric")
model.compile(
optimizer="adam",
loss="sparse_categorical_crossentropy",
metrics=[tf.keras.metrics.Precision()],
)
model.fit(x, y, epochs=2)
The output is
Metal device set to: Apple M1 Pro
2022-09-20 18:47:20.254419: I tensorflow/core/common_runtime/pluggable_device/pluggable_device_factory.cc:305] Could not identify NUMA node of platform GPU ID 0, defaulting to 0. Your kernel may not have been built with NUMA support.
2022-09-20 18:47:20.254522: I tensorflow/core/common_runtime/pluggable_device/pluggable_device_factory.cc:271] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 0 MB memory) -> physical PluggableDevice (device: 0, name: METAL, pci bus id: <undefined>)
2022-09-20 18:47:20.324585: W tensorflow/core/platform/profile_utils/cpu_utils.cc:128] Failed to get CPU frequency: 0 Hz
Train without precision metric
Epoch 1/2
2022-09-20 18:47:20.441786: I tensorflow/core/grappler/optimizers/custom_graph_optimizer_registry.cc:113] Plugin optimizer for device_type GPU is enabled.
1/1 [==============================] - ETA: 0s - loss: 5.9380
1/1 [==============================] - 0s 205ms/step - loss: 5.9380
Epoch 2/2
1/1 [==============================] - ETA: 0s - loss: 5.8844
1/1 [==============================] - 0s 4ms/step - loss: 5.8844
Train with precision metric
Epoch 1/2
systemMemory: 16.00 GB
maxCacheSize: 5.33 GB
Traceback (most recent call last):
File "/Users/dima/dev/learn/datascience/test-sparse-precision.py", line 35, in <module>
model.fit(x, y, epochs=2)
File "/Users/dima/sw/mambaforge/envs/data-science/lib/python3.10/site-packages/keras/utils/traceback_utils.py", line 67, in error_handler
raise e.with_traceback(filtered_tb) from None
File "/var/folders/_0/2yc8qfs11xq2vykxzkkngq4m0000gn/T/__autograph_generated_filedw4nh8_p.py", line 15, in tf__train_function
retval_ = ag__.converted_call(ag__.ld(step_function), (ag__.ld(self), ag__.ld(iterator)), None, fscope)
ValueError: in user code:
File "/Users/dima/sw/mambaforge/envs/data-science/lib/python3.10/site-packages/keras/engine/training.py", line 1051, in train_function *
return step_function(self, iterator)
File "/Users/dima/sw/mambaforge/envs/data-science/lib/python3.10/site-packages/keras/engine/training.py", line 1040, in step_function **
outputs = model.distribute_strategy.run(run_step, args=(data,))
File "/Users/dima/sw/mambaforge/envs/data-science/lib/python3.10/site-packages/keras/engine/training.py", line 1030, in run_step **
outputs = model.train_step(data)
File "/Users/dima/sw/mambaforge/envs/data-science/lib/python3.10/site-packages/keras/engine/training.py", line 894, in train_step
return self.compute_metrics(x, y, y_pred, sample_weight)
File "/Users/dima/sw/mambaforge/envs/data-science/lib/python3.10/site-packages/keras/engine/training.py", line 987, in compute_metrics
self.compiled_metrics.update_state(y, y_pred, sample_weight)
File "/Users/dima/sw/mambaforge/envs/data-science/lib/python3.10/site-packages/keras/engine/compile_utils.py", line 501, in update_state
metric_obj.update_state(y_t, y_p, sample_weight=mask)
File "/Users/dima/sw/mambaforge/envs/data-science/lib/python3.10/site-packages/keras/utils/metrics_utils.py", line 70, in decorated
update_op = update_state_fn(*args, **kwargs)
File "/Users/dima/sw/mambaforge/envs/data-science/lib/python3.10/site-packages/keras/metrics/base_metric.py", line 140, in update_state_fn
return ag_update_state(*args, **kwargs)
File "/Users/dima/sw/mambaforge/envs/data-science/lib/python3.10/site-packages/keras/metrics/metrics.py", line 818, in update_state **
return metrics_utils.update_confusion_matrix_variables(
File "/Users/dima/sw/mambaforge/envs/data-science/lib/python3.10/site-packages/keras/utils/metrics_utils.py", line 619, in update_confusion_matrix_variables
y_pred.shape.assert_is_compatible_with(y_true.shape)
ValueError: Shapes (None, 2) and (None, 1) are incompatible
It occurs on two different environments: Tensorflow 2.9.2 from Apple for M1, and on Tensorflow 2.8.0 on Ubuntu.
Does anyone know how to deal with this besides writing my own metric class?