How to filter dataset by tensor shape in Tensorflow

user_15853949 0
Viewed 532

I have loaded a dataset from tfds.load and want to throw away certain images that interfere with proper training/are of no use to me (for example, are too small).

It seems like there is absolutely no information on this specific problem anywhere so I went with what seems like the best fit which was .filter(predicate) on the dataset. Unfortunately the input to the predicate has indeterminate shape (None, None, 3) and as expected raises an error that 'int' cannot be compared with 'NoneType'.

Is it even possible to solve this problem in tensorflow or should I not waste my time?

Pseudo code

ds_train = tfds.load('name')
ds_train = ds_train.map(lambda ds: ds['image'])
ds_train = ds_train.filter(lambda image: image.shape[0] >= 256)
data-processing tensorflow2.0 tensorflow python
1 Answers

When writing code with tf.data.Dataset, you should use tf.shape(tensor) rather than tensor.shape, because tf.data.Dataset works in graph mode.

Quoting the documentation of tf.shape:

tf.shape and Tensor.shape should be identical in eager mode. Within tf.function or within a compat.v1 context, not all dimensions may be known until execution time. Hence when defining custom layers and models for graph mode, prefer the dynamic tf.shape(x) over the static x.shape.

ds_train = ds_train.filter(lambda image: tf.shape(image)[0] >= 256)
user_73701530
Related