What's the difference between using Dataset and ndarray in fit method in Tensorflow 2?

user_15024599 0
Viewed 824

As a newbie for TF, I feel a little confused about the usage of BatchDataset in training a model.

Let's use the MNIST as an example. In this classification task, we can load the data and feed the ndarray of x_trian, y_train directly into the model.

mnist = tf.keras.datasets.mnist

(x_train, y_train), (x_test, y_test) = mnist.load_data()
x_train, x_test = x_train / 255.0, x_test / 255.0

model = tf.keras.models.Sequential([
  tf.keras.layers.Flatten(input_shape=(28, 28)),
  tf.keras.layers.Dense(128, activation='relu'),
  tf.keras.layers.Dropout(0.2),
  tf.keras.layers.Dense(10, activation='softmax')
])

model.compile(optimizer='adam',
              loss='sparse_categorical_crossentropy',
              metrics=['accuracy'])

model.fit(x_train,y_train, epochs=5)

The training results are:

Epoch 1/5
2021-02-17 15:43:02.621749: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cublas64_10.dll
   1/1875 [..............................] - ETA: 0s - loss: 2.2977 - accuracy: 0.0938WARNING:tensorflow:Callbacks method `on_train_batch_end` is slow compared to the batch time (batch time: 0.0000s vs `on_train_batch_end` time: 0.0010s). Check your callbacks.
1875/1875 [==============================] - 2s 1ms/step - loss: 0.3047 - accuracy: 0.9117
Epoch 2/5
1875/1875 [==============================] - 2s 1ms/step - loss: 0.1473 - accuracy: 0.9569
Epoch 3/5
1875/1875 [==============================] - 2s 1ms/step - loss: 0.1097 - accuracy: 0.9673
Epoch 4/5
1875/1875 [==============================] - 2s 1ms/step - loss: 0.0905 - accuracy: 0.9724
Epoch 5/5
1875/1875 [==============================] - 2s 1ms/step - loss: 0.0759 - accuracy: 0.9764

And we can also use tf.data.Dataset.from_tensor_slices to generate a BatchDataset and feed it in to fit function.

mnist = tf.keras.datasets.mnist

(x_train, y_train), (x_test, y_test) = mnist.load_data()
x_train, x_test = x_train / 255.0, x_test / 255.0

train_ds = tf.data.Dataset.from_tensor_slices(
    (x_train, y_train)).shuffle(10000).batch(32)

test_ds = tf.data.Dataset.from_tensor_slices((x_test, y_test)).batch(32)

model = tf.keras.models.Sequential([
  tf.keras.layers.Flatten(input_shape=(28, 28)),
  tf.keras.layers.Dense(128, activation='relu'),
  tf.keras.layers.Dropout(0.2),
  tf.keras.layers.Dense(10, activation='softmax')
])

model.compile(optimizer='adam',
              loss='sparse_categorical_crossentropy',
              metrics=['accuracy'])

model.fit(train_ds, epochs=5)

The results in training process is as follows.

Epoch 1/5
2021-02-17 15:30:34.698718: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cublas64_10.dll
1875/1875 [==============================] - 3s 1ms/step - loss: 0.2969 - accuracy: 0.9140
Epoch 2/5
1875/1875 [==============================] - 3s 1ms/step - loss: 0.1462 - accuracy: 0.9566
Epoch 3/5
1875/1875 [==============================] - 3s 1ms/step - loss: 0.1087 - accuracy: 0.9669
Epoch 4/5
1875/1875 [==============================] - 3s 1ms/step - loss: 0.0881 - accuracy: 0.9730
Epoch 5/5
1875/1875 [==============================] - 3s 1ms/step - loss: 0.0765 - accuracy: 0.9759

The model can be trained successfully with 2 methods, but is there any difference between them? Does using Dataset for training have some additional advantages? If there is no difference between the 2 methods in this case, what the typical usage of generating a Dataset for training and when should this method be used?

Thank you.

keras tensorflow
1 Answers

When we use Model.fit(x=None, y=None, ... - we can pass the training pair argument as pure numpy array or keras.utils.Sequence or tf.data.

When we use as follows, we're passing each training pairs (x and y) separately as a direct numpy array to the fit function.

# data 
(x_train, y_train), (_, _) = tf.keras.datasets.mnist.load_data()

# fit
model.fit(x = x_train, y = y_train, ... 

# check
print(x_train.shape, y_train.shape)
print(type(x_train), type(y_train))

# (60000, 28, 28) (60000,)
# <class 'numpy.ndarray'> <class 'numpy.ndarray'>

On the other hand in tf.data and Sequence we pass the training pairs as a shape of the tuple and still the data type are ndarray. According to the doc,

  • A tf.data dataset. Should return a tuple of either (inputs, targets)
  • A generator or keras.utils.Sequence returning (inputs, targets)

i.e

# data
train_ds = tf.data.Dataset.from_tensor_slices((x_train, y_train)).shuffle(10000).batch(2)

# check
next(iter(train_ds))

(<tf.Tensor: shape=(2, 28, 28), dtype=uint8, numpy= array([[[...], [[...]]], dtype=uint8)>,
 <tf.Tensor: shape=(2,), dtype=uint8, numpy=array([7, 8], dtype=uint8)>)

And that's why, if x is a tf.data, generator, or keras.utils.Sequence instance, y should not be specified (since targets will be obtained from x).

# fit 
model.fit(train_ds, ...

Among these three, tf.data data pipelines is the most efficient approach followed by generator. When the data set is small enough, the first approach (x and y) is primarily chosen. But when the dataset gets big enough, then you would think about tf.data or generator for efficient input pipelines. So the choice of these totally depends.

From Keras's post:

  • NumPy arrays, just like Scikit-Learn and many other Python-based libraries. This is a good option if your data fits in memory.

  • TensorFlow Dataset objects. This is a high-performance option that is more suitable for datasets that do not fit in memory and that are streamed from disk or from a distributed filesystem.

  • Python generators that yield batches of data (such as custom subclasses of the keras.utils.Sequence class).

user_92157800
Related