AssertionError when using MirroredStrategy: isinstance(x, dataset_ops.DatasetV2)

user_5924132 0
Viewed 1807

I am trying to use MirroredStrategy to fit my sequential model using two Titan Xp GPUs. I am using tensorflow 2.0 alpha on ubuntu 16.04.

I successfully run the code snippet from the tensorflow documentation:

from __future__ import absolute_import, division, print_function, unicode_literals
import tensorflow as tf

mirrored_strategy = tf.distribute.MirroredStrategy()
  with mirrored_strategy.scope():
  model = tf.keras.Sequential([tf.keras.layers.Dense(1, input_shape=(1,))])
  model.compile(loss='mse', optimizer='sgd')

dataset = tf.data.Dataset.from_tensors(([1.], [1.])).repeat(100).batch(10)
model.fit(dataset, epochs=2)
model.evaluate(dataset)

However, when I try to train on my data, which is a sparse matrix of shape (using adam optimizer and binary crossentropy):

Shape X_train: (91422, 65545)
Shape y_train: (91422, 1)

I receive an assertion error in _distribution_standardize_user_data at

assert isinstance(x, dataset_ops.DatasetV2)

In the TensorFlow code, line 2166 in training.py seems to be causing this assertion error.

Can someone explain to me what the problem with my data could be?

tensorflow assertion gpu python
2 Answers

I got similar error when using dataset= strategy.experimental_distribute_dataset(train_dataset) with model.fit(dataset) .

I after I remove the strategy.experimental_distribute_dataset. It works fine. It is similar to the TF document where they said that keras.Model.fit() handle everything automatically and we need distributed dataset manually only when we want to do customized training with tf.GradientTape().

You can go through the offical tutorial of MNIST for more info

user_91372190

Seems like you are feed dataset into model.fit, model.fit are expecting an numpy.ndarray.

user_114683020
Related