How to get validation set which has equal number of images for each class using tensorflow?

user_17574229 0
Viewed 275

I'm now using CIFAR-100 dataset to train a model. I'd like to use 10% of train data as validation data. I used the codes below in the beginning.

(train_images, train_labels), (test_images, test_labels) = datasets.cifar100.load_data()
train_images, val_images, train_labels, val_labels = train_test_split(train_images, train_labels, test_size=0.1)

train_db = tf.data.Dataset.from_tensor_slices((train_images, train_labels))
train_db = train_db.map(train_prep).shuffle(5000).repeat().batch(128).prefetch(-1)

val_db = tf.data.Dataset.from_tensor_slices((val_images, val_labels))
val_db = val_db.map(valid_prep).batch(512).prefetch(-1)

It works good in some models. But in some other models, the validation accuracy can be way higher than the test accuracy. I think the reason may be using train_test_split could not guarantee that validation set has same number of images each class. So I tried to get validation set 'manually'. My codes are shown below.

(train_images, train_labels), (test_images, test_labels) = datasets.cifar100.load_data()

def get_index(y):
  index = [[] for i in range(100)]
  for i in range(len(y)):
      for j in range(100):
          if y[i][0] == j:
              index[j].append(i)
  return index

index = get_index(train_labels)

index_train = []
index_val = []
for i in range(100):
  index1, index2 = train_test_split(index[i], test_size=0.1)
  index_train.extend(index1)
  index_val.extend(index2)

val_images = train_images[index_val]
train_images_1 = train_images[index_train]

val_labels = train_labels[index_val]
train_labels_1 = train_labels[index_train]

train_db = tf.data.Dataset.from_tensor_slices((train_images_1, train_labels_1))
train_db = train_db.map(train_prep).shuffle(5000).repeat().batch(128).prefetch(-1)

val_db = tf.data.Dataset.from_tensor_slices((val_images, val_labels))
val_db = val_db.map(valid_prep).batch(512).prefetch(-1)

But when I use this training set and validation set to train my models, the accuracy is rather low. So there must be some problems of this splitting method. But I don't know what's the problem. I'd be appreciate if someone could help me with this problem.

data-preprocessing tensorflow validation python
2 Answers

train_test_split has a parameter called stratify that may help you. In the example below assume the data frame df, has 2 columns. One is called filepaths and in it each row of the column contains the full path to an image file. The second column is called labels. Each row in the column contains text which identifies what class the image in that row is. For example if you are classifying images of dogs and cats then the labels would be "dog" or "cat". Assume that 80% of the images are of cats and 20% are of dogs. When you split the data set you can ensure that the split is done such that the resulting data frame each have 80% cat images and 20% dog images. The code for that is

train_split=.8 # % of images to use for training
validation_split=.1 # % of images to use for validation
dsplit=validation_split/(1-train_split)
train_ds, dummy_df=train_test_split(df, train_size=train_split, shuffle=True, random_state=123, stratify=df['labels'])
valid_df, test_df=train_test_split(dummy_df, train_size=dsplit, shuffle=True, random_state=123, stratify=df['labels'])

the result is 3 datasets train_df, test_df and valid_df. Each data set will have the same ratio of cat and dog class labels as the original dataframe df. Now to get a balanced data set where 50% of the labels are of cats and 50% of the labels are of dogs you need to do under sampling, image augmentation or a combination of both.

user_107989170

I had a similar problem. I wanted to split my data equally for each class but with keeping the order of the data in input. In other words: So I made a function to do it.
Note that the data is not shuffle before splitting, but after splitting.

def unison_shuffled_copies(a, b):
    assert len(a) == len(b)
    p = np.random.permutation(len(a))
    return a[p], b[p]

def split_dataset(X, y, size=0.5, shuffle_a = True):
    ax = []
    bx = []
    ay = []
    by = []
    # Iterate over the labels
    for label in np.unique(y):
        count = 0
        # Calculate the maximum number of values to include
        max_count = len(y[y == label]) * size
        for i in range(0,len(y)):
            if y[i] == label: # Make sure we treat only a specific label
                if (count < max_count):
                    ax.append(X[i])
                    ay.append(y[i])
                else:
                    bx.append(X[i])
                    by.append(y[i])
                count += 1
                        
    if shuffle_a:
        ax, ay = unison_shuffled_copies(np.array(ax), np.array(ay))
    else:
        ax, ay = (np.array(ax), np.array(ay))
    bx, by = unison_shuffled_copies(np.array(bx), np.array(by))
    return ax, bx, ay, by

If you don't want a proportion but a specific number, then you can directly replace :

max_count = len(y[y == label]) * size

by

max_count = size

or add a max_count parameter.

Note that the size percentage is reversed compared to the tensorflow functions, it represent the size of A and not the size of B.

Disclaimer

There are better ways to implement this split function. If you copy it, let it be for test purposes or dataset with few categories. If you want to improve the performance, instead of running the loop once per label, create dict with the labels as key and the count and maximum counts values, then run the loop only once.

user_179653130
Related