How tensorflow knows which part of data are assigned to which subdataset?

user_2280637 0
Viewed 57

The code snippet is copied from TensorFlow's tutorial website (link). There are two code blocks, one for train_ds and the other for val_ds. They are identical except for the the subset= argument. I am wondering whether TensorFlow assigns the first 80% of the data to train_ds and the rest of the data to val_ds. If not, how does TensorFlow know which part is assigned to which? Thanks.

train_ds = tf.keras.preprocessing.image_dataset_from_directory(
  data_dir,
  validation_split=0.2,
  subset="training",
  seed=123,
  image_size=(img_height, img_width),
  batch_size=batch_size
)
val_ds = tf.keras.preprocessing.image_dataset_from_directory(
  data_dir,
  validation_split=0.2, #L: The same as above
  subset="validation",
  seed=123,
  image_size=(img_height, img_width),
  batch_size=batch_size
)
keras tensorflow
1 Answers

We can look at the source code.

It basically boils down to that function:

def get_training_or_validation_split(samples, labels, validation_split, subset):
  """Potentially restict samples & labels to a training or validation split.
  Args:
    samples: List of elements.
    labels: List of corresponding labels.
    validation_split: Float, fraction of data to reserve for validation.
    subset: Subset of the data to return.
      Either "training", "validation", or None. If None, we return all of the
      data.
  Returns:
    tuple (samples, labels), potentially restricted to the specified subset.
  """
  if not validation_split:
    return samples, labels

  num_val_samples = int(validation_split * len(samples))
  if subset == 'training':
    print('Using %d files for training.' % (len(samples) - num_val_samples,))
    samples = samples[:-num_val_samples]
    labels = labels[:-num_val_samples]
  elif subset == 'validation':
    print('Using %d files for validation.' % (num_val_samples,))
    samples = samples[-num_val_samples:]
    labels = labels[-num_val_samples:]
  else:
    raise ValueError('`subset` must be either "training" '
                     'or "validation", received: %s' % (subset,))
  return samples, labels

The training set uses the first part of the samples, while the validation set uses the last part.

user_73701530
Related