Is Tensorflow Dataset API slower than Queues?

user_867889 0
Viewed 2883

I replaced CIFAR-10 preprocessing pipeline in the project with Dataset API approach and it resulted in performance decrease of about 10-20%.

Preporcessing is rather standart: - read image from disk - make random/crop and flip - shuffle, batch - feed to the model

Overall i see that batche processing is now 15% faster, but every once in a while (or, more precisely, whenever I reinitialize dataframe or expect reshuffling) the batch is being blocked for up long time (30 sec) which totals to slower epoch-per-epoch processing.

This behaviour seems to do something with internal hashing. If I reduce N in ds.shuffle(buffer_size=N) delays are shorter but proportionally more frequent. Removing shuffle at all results to delays as if buffer_size was set to dataset size.

Can somebody explain internal logic of Dataset API when it comes to reading/caching? Is there any reason at all to expect Dataset API to work faster than manually created Queues?

I am using TF 1.3.

tensorflow-datasets tensorflow performance
2 Answers

Suggested things didn't solve my problem back in the days, but I would like to add a couple of recommendations for those, who don't want to learn about queues and still get the most out of TF data pipeline:

.

files = tf.data.Dataset.list_files(data_dir)
ds = tf.data.TFRecordDataset(files, num_parallel_reads=32)
ds = (ds.shuffle(10000)
    .repeat(EPOCHS)
    .map(parser_fn, num_parallel_calls=64)
    .batch(batch_size)
)
dataset = dataset.prefetch(2)

Where you have to pay attention to 3 main components:

  • num_parallel_read=32 to parallelize disk IO operations
  • num_parallel_calls=64 to parallelize calls to parser function
  • prefetch(2)
user_8678890
Related