difference between tf.data.Dataset batch and map and tf.contrib.data.map_and_batch

user_9637085 0
Viewed 2727

I created a tf.data.Dataset and want to train a model using this dataset:

dataset = dataset.prefeth()
dataset = dataset.shuffle()
dataset = dataset.repeat()
dataset = dataset.map()
dataset = dataset.filter()
dataset = dataset.batch()

I want to know what is the difference between the above dataset with the bellow one:

dataset = dataset.prefeth()
dataset = dataset.shuffle()
dataset = dataset.repeat()
dataset = dataset.apply(tf.contrib.data.map_and_batch())

I know that they should not be different except in performance. But I don't know should I use the .apply() method or not?

Is the first implementation correct?

tensorflow-estimator tensorflow-datasets tensorflow python-3.x
1 Answers

First off, most of the tf.contrib.data functions are deprecated and moved to tf.data.experimental. So watch out for that.

Take a look at input pipeline performance guide to get a good idea about what could be a good optimal ordering of the transformations for your application. Regarding map and batch, yes we pass the result of the map and batch to the apply function, and it is specified in the return description of map and batch for reference confirmation. And we want to use map and batch for efficiency reasons, which normally depends on what your data is and how costly your map function is. The performance guide has some guidelines for the same.

Regarding the difference between your first and second blocks of code, there is a filter function in between, so both blocks might not give the same result depending on what you are filtering.

user_101119310
Related