I am working with Tensorflow input data pipeline on a classification problem with 3 classes. I am trying to create balanced batch and n0=100,n1=10 and n2=20 are selected based on relative size of each class in dataset. However, I get this error:
2021-12-26 14:14:24.168439: W tensorflow/core/kernels/data/cache_dataset_ops.cc:768] The calling iterator did not fully read the dataset being cached.
In order to avoid unexpected truncation of the dataset, the partially cached contents of the dataset will be discarded.
This can happen if you have an input pipeline similar to`dataset.cache().take(k).repeat()`. You should use `dataset.take(k).cache().repeat()` instead.
This is my code:
Train_dataset = tf.data.Dataset.list_files(file_list)
Train_dataset= Train_dataset.interleave(lambda filename: tf.data.TextLineDataset(filename))
Train_dataset= Train_dataset.map(separate_input_output)
class0 = Train_dataset.filter(lambda x,y: y==0).cache().repeat(n0*nepocs)
class1 = Train_dataset.filter(lambda x,y: y == 1).cache().repeat(n1*nepocs)
class2 = Train_dataset.filter(lambda x,y: y == 2).cache().repeat(n2*nepocs)
Train_dataset = tf.data.Dataset.zip((class0, class1, class2))
Train_dataset=Train_dataset.flat_map(lambda c0, c1, c2:tf.data.Dataset.from_tensors(c0).concatenate(tf.data.Dataset.from_tensors(c1).concatenate(tf.data.Dataset.from_tensors(c2))))
Train_dataset= Train_dataset.batch(64,drop_remainder=True).prefetch(buffer_size=1)
Does anyone know why I can not fully read the dataset being cached?