as_list() is not defined on an unknown TensorShape on y_t_rank = len(y_t.shape.as_list()) and related to metrics

user_1762295 0
Viewed 2083

TF 2.3.0.dev20200620

I got this error during .fit(...) for a model with a sigmoid binary output. I used tf.data.Dataset as the input pipeline. The strange thing is it depends on the metric:

Don't work:

model.compile(
    optimizer=tf.keras.optimizers.Adam(lr=1e-4, decay=1e-6),
    loss=tf.keras.losses.BinaryCrossentropy(),
    metrics=['accuracy']
)

work:

model.compile(
    optimizer=tf.keras.optimizers.Adam(lr=1e-4, decay=1e-6),
    loss=tf.keras.losses.BinaryCrossentropy(),
    metrics=[tf.keras.metrics.BinaryAccuracy()]
)

But as I understood, 'accuracy' should be fine. In fact, instead of using my own tf.data.Dataset custom setup (can be provided if needed), using tf.keras.preprocessing.image_dataset_from_directory give no such error. This is the case from tutorial https://keras.io/examples/vision/image_classification_from_scratch.

Trace is pasted below. Notice this is diff from other 2 older questions. it involves somehow the metrics.

ValueError: in user code:

/usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/engine/training.py:806 train_function  *
    return step_function(self, iterator)
/usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/engine/training.py:796 step_function  **
    outputs = model.distribute_strategy.run(run_step, args=(data,))
/usr/local/lib/python3.6/dist-packages/tensorflow/python/distribute/distribute_lib.py:1211 run
    return self._extended.call_for_each_replica(fn, args=args, kwargs=kwargs)
/usr/local/lib/python3.6/dist-packages/tensorflow/python/distribute/distribute_lib.py:2526 call_for_each_replica
    return self._call_for_each_replica(fn, args, kwargs)
/usr/local/lib/python3.6/dist-packages/tensorflow/python/distribute/distribute_lib.py:2886 _call_for_each_replica
    return fn(*args, **kwargs)
/usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/engine/training.py:789 run_step  **
    outputs = model.train_step(data)
/usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/engine/training.py:759 train_step
    self.compiled_metrics.update_state(y, y_pred, sample_weight)
/usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/engine/compile_utils.py:388 update_state
    self.build(y_pred, y_true)
/usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/engine/compile_utils.py:319 build
    self._metrics, y_true, y_pred)
/usr/local/lib/python3.6/dist-packages/tensorflow/python/util/nest.py:1139 map_structure_up_to
    **kwargs)
/usr/local/lib/python3.6/dist-packages/tensorflow/python/util/nest.py:1235 map_structure_with_tuple_paths_up_to
    *flat_value_lists)]
/usr/local/lib/python3.6/dist-packages/tensorflow/python/util/nest.py:1234 <listcomp>
    results = [func(*args, **kwargs) for args in zip(flat_path_list,
/usr/local/lib/python3.6/dist-packages/tensorflow/python/util/nest.py:1137 <lambda>
    lambda _, *values: func(*values),  # Discards the path arg.
/usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/engine/compile_utils.py:419 _get_metric_objects
    return [self._get_metric_object(m, y_t, y_p) for m in metrics]
/usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/engine/compile_utils.py:419 <listcomp>
    return [self._get_metric_object(m, y_t, y_p) for m in metrics]
/usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/engine/compile_utils.py:440 _get_metric_object
    y_t_rank = len(y_t.shape.as_list())
/usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/tensor_shape.py:1190 as_list
    raise ValueError("as_list() is not defined on an unknown TensorShape.")

ValueError: as_list() is not defined on an unknown TensorShape.
tensorflow-datasets keras tensorflow
2 Answers

Had exactly the same problem when using 'accuracy' metric.

I followed https://github.com/tensorflow/tensorflow/issues/32912#issuecomment-550363802 example:

def _fixup_shape(images, labels, weights):
    images.set_shape([None, None, None, 3])
    labels.set_shape([None, 19]) # I have 19 classes
    weights.set_shape([None])
    return images, labels, weights
dataset = dataset.map(_fixup_shape)

which helped me solve the problem.

But, in my case, instead of using one map function, as kawingkelvin did above, to load and set_shape inside, I needed to use two map functions because of some errors in the TF code.

The final solution for me was to use the following order: dataset.batch.map(get_data).map(fix_shape).prefetch

NOTE: batch can be done both before and after map(get_data) depending on how your get_data function is created. Fix_shape must be done after.

user_55363880

I am able to fix this in such a way as to keep the metrics 'accuracy' (rather than using BinaryAccuracy). However, I do not quite understand why this is needed for 'accuracy', but not needed for other closely related one (e.g. BinaryAccuracy).

2 things:

  1. construct a ds such that the batch label has shape of (batch_size, 1) but not (batch_size,). Following the keras.io tutorial mentioned, it should have been ok with the latter. This change aims to get rid of the "unknown" in the TensorShape.

  2. add this to the ds pipeline:

    label.set_shape([1])

     def process_path(file_path):
   label = get_label(file_path)
   img = tf.io.read_file(file_path)
   img = tf.image.decode_jpeg(img, channels=3)

   label.set_shape([1])

   return img, label

 ds = ds.map(process_path, num_parallel_calls=AUTO).shuffle(1024).repeat().batch(batch_size).prefetch(buffer_size=AUTO)

This is the state before .batch(...) so a single sample should have 1 as the shape (and therefore, (batch_size, 1) after batching.

After doing so, the error didn't happen, and I used the exact same metrics 'accuracy' as in https://keras.io/examples/vision/image_classification_from_scratch

Hope this helps anyone who got hit. I have to admit I don't truly understand why it didn't work in the first place. It still seems like a TF bug to me.

user_17622950
Related