Tensorflow: concatenating multiple tf.Dataset very slow

Viewed 948

I am on Tensorflow 1.10

Right now I am not sure if this is a bug.

I have been trying to concatenate about 100 Datasets which I generated from multiple tf.data.Dataset.from_generator.

for i in range(1, 100):
        dataset = dataset.concatenate(
            tf.data.Dataset.from_generator(gens[i], (tf.int8, tf.int32), output_shapes=(
                (256, 256), (1))))
        print(i)
 print("before iterator")
 iterator = dataset.make_one_shot_iterator()
 print("after iterator")

running the make_one_shot_iterator() takes really long.

Anyone knows a fix?

EDIT:

It looks like that _make_dataset.add_to_graph(ops.get_default_graph()) seems to get called over and over again resulting in a few million calls of the function. (https://github.com/tensorflow/tensorflow/blob/master/tensorflow/python/data/ops/dataset_ops.py function make_one_shot_iterator line 162)

1 Answers
Related