I am on Tensorflow 1.10
Right now I am not sure if this is a bug.
I have been trying to concatenate about 100 Datasets which I generated from multiple tf.data.Dataset.from_generator.
for i in range(1, 100):
dataset = dataset.concatenate(
tf.data.Dataset.from_generator(gens[i], (tf.int8, tf.int32), output_shapes=(
(256, 256), (1))))
print(i)
print("before iterator")
iterator = dataset.make_one_shot_iterator()
print("after iterator")
running the make_one_shot_iterator() takes really long.
Anyone knows a fix?
EDIT:
It looks like that _make_dataset.add_to_graph(ops.get_default_graph())
seems to get called over and over again resulting in a few million calls of the function.
(https://github.com/tensorflow/tensorflow/blob/master/tensorflow/python/data/ops/dataset_ops.py function make_one_shot_iterator line 162)