GeneratorDatasetOp:Dataset will not be optimized because the dataset does not implement the AsGraphDefInternal() method needed to apply optimizations

Question

I have created a tf record and used it to train my model. During the training process, Whenever I am calling my data to perform the evaluation I am always getting the message

Input of GeneratorDatasetOp::Dataset will not be optimized because the dataset does not implement the AsGraphDefInternal() method needed to apply optimizations.

During the creation of tf record, I try to include below code to optimize the dataset but this doesn't worked out

AUTO = tf.data.experimental.AUTOTUNE # used in tf.data.Dataset API
option_no_order = tf.data.Options()
option_no_order.experimental_deterministic = False

option_no_order.experimental_optimization.noop_elimination = True
option_no_order.experimental_optimization.apply_default_optimizations = True

Please suggest the steps that I can try out or follow to optimize the dataset

user_11836236 · Answer

I got the same warning message when my model was created using distributed strategy. Creating the model with non-distributed strategy removed the warning. If you need to use distributed strategy, I think you need to use distributed dataset as well (https://www.tensorflow.org/api_docs/python/tf/distribute/DistributedDataset).

GeneratorDatasetOp:Dataset will not be optimized because the dataset does not implement the AsGraphDefInternal() method needed to apply optimizations

1 Answers