tf.Data.Dataset - On each Epoch, only train with a sub sample of the full dataset

user_2252698 0
Viewed 657

I have an image dataset with a large imbalance of positive and negatives samples (many more negatives). I would like to create a tf.data.Dataset where each epoch it will train with all of the positive samples but only (ratio * len(positive) ) of the negative samples.

I am currently using a datagen inherited from keras.util.Sequence to achieve this and using this subsampling policy is performing much better than training on all data.

However reading the docs on Dataset, I cannot seem to find a way to do it, is it possible?

In my existing data generator, I am doing this:

# List if indicies of the positive and negative samples
positives = np.where(self.labels == 1)[0]
negatives = np.where(self.labels == 0)[0]
# How many of the negatives do we want to use?
n_negatives = np.clip(int(len(positives) * self.config.DATASET_NEGSUBSAMPLE_RATIO), 1, len(negatives))
# Choose random negatives
subsampled_negatives = np.random.choice(negatives, n_negatives, replace=False)
# Create the incidies array from the positive and subsamples negative indicies
self.indexes = np.concatenate((positives, subsampled_negatives))
# Shuffle them together
np.random.shuffle(self.indexes)
tensorflow python
1 Answers

positives and negatives are defined as in the question.

positives = [(0,1),(1,1),(2,1),(3,1),(4,1)]
negatives = [(10,0),(11,0),(12,0),(13,0),(14,0),(15,0),(16,0)]

NEGATIVE_SAMPLES = 3

pos_ds = tf.data.Dataset.from_tensor_slices(positives)
neg_ds = tf.data.Dataset.from_tensor_slices(negatives).shuffle(1000)

ds = pos_ds.concatenate(neg_ds.take(NEGATIVE_SAMPLES)).shuffle(1000)

els = [v.numpy().tolist() for v in list(ds)]

printing els for a sample execution gives:

[[0, 1], [4, 1], [12, 0], [16, 0], [1, 1], [10, 0], [3, 1], [2, 1]]

Note: you might need to experiment with the buffer sizes for the shuffle at the end of the ds definition.

user_32148720
Related