splitting dataset before generating TensorFlow dataset

Viewed 83

If I generate a tf.dataset from the whole data and then split it to training, test, and val sets, the training works fine and loss goes down on both training and validation sets.

But, if I first split my data to training, test, and val and then generate a tf.dataset for each of them, the loss goes down on the training set but not on the validation set.

I do not know why there is a difference between these two scenarios. I was wondering if somebody could explain it to me. Thanks

def gen_dataset(molset, n_feat):
  def generator():
    for mol in molset:
      X = mol.feat()
      y = mol.value()
      yield X, y

  dataset = tf.data.Dataset.from_generator(generator,
            output_types=((tf.float32, tf.float32),tf.float32),
            output_shapes=((tf.TensorShape([None,n_feat]),
            tf.TensorShape([None, n_feat])),
            tf.TensorShape([])))

  return dataset

#In scenario 1, tf.data.Dataset.from_generator is called on data and then the dataset is split to training, test, and val

dataset  = gen_dataset(molset=data, n_feat)

training = dataset.take(training_size)
test     = dataset.skip(test_size).take(test_size)
val      = dataset.skip(valid_size).take(valid_size)

#In scenario 2, data is first split to test_set, train_set, val_set and then tf.data.Dataset.from_generator is called on each of them

training = gen_dataset(molset=train_set, n_feat)  
test     = gen_dataset(molset=test_set,  n_feat) 
val      = gen_dataset(molset=val_set,   n_feat)
1 Answers

Hard to say without more details about the code. Investigate to make sure that samples are not leaking from train into test in the first case; it could be that your model is fitting to validation samples by accident and so it erroneously is showing progress, whereas this doesn't happen in case 2 and so it seems like something is wrong. Maybe post a Google Colab to demonstrate the problem.

Related