I have a TF dataset
train_ds = train_ds.cache().shuffle(1000).prefetch(buffer_size=AUTOTUNE)
val_ds = val_ds.cache().prefetch(buffer_size=AUTOTUNE)
And since my data is huge (100,000) I want to select only a subset of data to train on So i need to create a new TF dataset from the old dataset
You could use tf.data.Dataset.shard.
Creates a Dataset that includes only 1/num_shards of this dataset.
shard is deterministic. The Dataset produced by A.shard(n, i) will contain all elements of A whose index mod n = i.
A = tf.data.Dataset.range(10)
B = A.shard(num_shards=3, index=0)
If you want 1/10 of your original train_ds:
new_ds = train_ds.shard(num_shards=10, index=0)