Split a Tensorflow Dataset into Train, Validation, Test sets, does this code cause data leakage?

Viewed 23

I imported a folder that contains 3 folders of images insidethe , each belonging to one class, suppose cats and dogs. I created TensorFlow dataset by (shuffle=True) :

ds = tf.keras.preprocessing.image_dataset_from_directory('/content/data',
                                                    labels='inferred',
                                                    label_mode='int',
                                                    batch_size=32,
                                                    image_size=(256, 256),
                                                    shuffle=True)

Suppose len(ds) = 10

is it a correct way of splitting ds to train, Val, and test datasets? I found it on some websites and youtube videos including towardsdatascience.com:

train_ds = ds.take(8)    
val_ds = ds.skip(8).take(1)
test_ds = ds.skip(8).skip(1)

my question is: when the shuffle=True in image_dataset_from_directory function, each time I call ds.take(n) (n could be any integer less than 10), it returns different outputs, so I conclude that: it is highly probable that some data from train_ds exists in val_ds and test_ds. I mean there is data leakage... Am I right?

1 Answers

Read the documentation below Doc

It suffle only shuffle your data if true , other rake it inaccending order, and tosplit data you can use your code , many people do this and this does not show data leakage

Related