I've got a custom TensorFlow Dataset and my problem is , that my Validation Dataset is loading indefinitely, if i try to access the first item.
So next(iter(train_ds.take(1))) returns the first Training-Data as expected, but next(iter(val_ds.take(1))) loads indefinitely.
My Dataset Contains multiple Image-Path-Triplets (<ZipDataset shapes: ((), (), ()), types: (tf.string, tf.string, tf.string)>).
My-Preprocessing looks something like this:
buffer_size = 1024
batch_size = 32
def prep_image(filename, img_shape=(144,144), channels=3, dtype=tf.float32):
image_string = tf.io.read_file(filename)
image = tf.image.decode_jpeg(image_string, channels=channels)
image = tf.image.convert_image_dtype(image, dtype)
image = tf.image.resize(image, img_shape)
return image
def preprocess_triplet(a, p, n):
return prep_image(a), prep_image(p), prep_image(n)
dataset = dataset.shuffle(buffer_size)
dataset = dataset.map(preprocess_triplet)
# splitting the ds
split = .8
n_train_items = round(len(dataset) * split) #r(514551 * 0.8) = 411641
train_dataset = dataset.take(n_train_items)
val_dataset = dataset.skip(n_train_items)
train_dataset = train_dataset.batch(batch_size, drop_remainder=False)
train_dataset = train_dataset.prefetch(tf.data.AUTOTUNE) #.prefetch(tf.data.AUTOTUNE)
val_dataset = val_dataset.batch(batch_size, drop_remainder=False)
val_dataset = val_dataset.prefetch(tf.data.AUTOTUNE)
When i run next(iter(train_dataset.take(1))) im getting my Image-Triplets as Tuple of Arrays (as expected). When i try to do the same on the Validation-Dataset it runs forever type(next(iter(val_dataset.take(1)))).
Verbose Version - Containing sanity Checks
python next(iter(dataset))
(<tf.Tensor: shape=(), dtype=string, numpy=b'F:\\workspace\\datasets\\own_256\\hm\\pics\\bf\\67\\bf67f3922e8245954943f1479961d932d56ef0ad.jpg'>,
<tf.Tensor: shape=(), dtype=string, numpy=b'F:\\workspace\\datasets\\own_256\\hm\\pics\\4b\\03\\4b03c337d013500930b71f9999766656f5893946.jpg'>,
<tf.Tensor: shape=(), dtype=string, numpy=b'F:\\workspace\\datasets\\own_256\\asos\\pics\\products\\asos-design-petite-halter-midi-beach-dress-in-double-gauze-in-sage-green\\24233155-2.jpg'>)
After Loading the Dataset and Shuffling
dataset = dataset.shuffle(buffer_size)
dataset = dataset.map(preprocess_triplet)
i just tried to "load" each entry, to check if the Images can be read:
for img in dataset:
a, b, c = img
assert a.shape == (144, 144, 3) and b.shape == (144, 144, 3) and c.shape == (144, 144, 3)
This run without any problems.
Afterwards i tried to Split the Dataset into Validation and Test
split = .8
n_train_items = round(len(dataset) * split) #r(514551 * 0.8) = 411641
train_dataset = dataset.take(n_train_items)
val_dataset = dataset.skip(n_train_items)
To test if i can access the Items i just called the first item of the batch per Val/Train DS:
next(iter(train_dataset))
$(<tf.Tensor: shape=(144, 144, 3), dtype=float32, numpy=
array(..., dtype=float32)>,
<tf.Tensor: shape=(144, 144, 3), dtype=float32, numpy=
array(..., dtype=float32)>,
<tf.Tensor: shape=(144, 144, 3), dtype=float32, numpy=
array(..., dtype=float32)>)
Also tried to iterate over all Train Data and it worked
print("t_items")
test_n_items = sum([sum([1 for item in a])for a, p, n in train_dataset])
print(test_n_items == n_train_items)
$True
But i cant even get the first item of Validation Dataset
next(iter(val_dataset))
Sadly the Second next(iter(val_dataset)) runs indefinitely.
Some Info
len(dataset)
$ 514551
len(train_dataset)
$411641
len(val_dataset)
$len(val_dataset)
len(dataset) == len(train_dataset) + len(val_dataset)
$True
Edit 1:
The Problem is withingprep_image, when i just return the Filename without opening the Data everything works as expected:
def prep_image(filename, img_shape=(144,144), channels=3, dtype=tf.float32):
return filename
# image_string = tf.io.read_file(filename)
# image = tf.image.decode_jpeg(image_string, channels=channels)
# image = tf.image.convert_image_dtype(image, dtype)
# image = tf.image.resize(image, img_shape)
# return image
Just reading the File:
def prep_image(filename, img_shape=(144,144), channels=3, dtype=tf.float32):
return tf.io.read_file(filename)
already produces the error of waiting for ever on Val-Dataset (Train still works).
Edit 2:
Simple Code to Reproduce the Issue: https://colab.research.google.com/drive/1w5RQDp6W3Zyu4p1kEBRlHT2Vl9zvxVOk?usp=sharing
Edit 3:
Loading the first Item within the Training-Dataset (from the example of ´Edit 2takes 0.6 msand loading the first item of the Validation Dataset takes31.09 Minutes`