way to train a model when we have a huge dataset so as to avoid error memory

Viewed 2115

I am trying to train a neural network with a dataset which have more thant 250k images. But I am stuck because of my limited computer which have 16 GB RAM and 32 GB SWAP. Before I load all the images, it gives me an memory error. So I was wondering if there is a way to train my neural network using all the images I have ? For example, instead of using RAM memory to load images in numpy array, can we load it on the free space disk ?

EDIT 1:

def get_array_image(file, path):
    return cv2.imread(path+file)
def generator(features, labels, num_classes, batch_size, path=''):
     # Create empty arrays to contain batch of features and labels#
     batch_features = np.zeros((batch_size, 28, 28, 3))
     batch_labels = np.zeros((batch_size, 1))
     while True:
       for cpt in range(0, len(features), batch_size):
         for i in range(0, batch_size):
             index = cpt + i
             #print('images : ', index)
             batch_features[i] = get_array_image(features[index], path)
             batch_labels[i] = labels[index]
         yield batch_features, keras.utils.to_categorical(batch_labels, num_classes)

This is my generator I used with fit_generator function. But I have a problem of accuracy. In fact, I tried it on the mnist dataset with a smal Neural Network. If I use the fit function loading all the images (about 60k images for training and 60k for the test) I have about 0.68 of accuracy after one epoch. But with the fit_generator I obtain only 0.1 . Am I doing something wrong with my generator ? When I print the index variable, it seems good.

EDIT 2: I solved my problem but i do not understand why it works. In fact when I create the array outside the loop I obtain low accuracy but when it is inside the accuracy is good with fit_generator. Does someone know what I am missing ?

def generator(features, labels, num_classes, batch_size, path='', dtype=np.uint8):
     # Create empty arrays to contain batch of features and labels#
     # batch_features = np.ndarray(shape=(batch_size, 28, 28, 3), dtype=dtype)
     # batch_labels =  np.ndarray(shape=(batch_size, 1), dtype=dtype)
     while True:
       for cpt in range(0, len(features), batch_size):

         batch_features = np.ndarray(shape=(batch_size, 28, 28, 3), dtype=dtype)
         batch_labels =  np.ndarray(shape=(batch_size, 1), dtype=dtype)
         for i in range(0, batch_size):
             # index= random.randint(0, len(features)-1)
             index = cpt + i
             #print('images : ', index)
             batch_features[i] = get_array_image(features[index], path)
             batch_labels[i] = labels[index]
#             print(batch_labels[i])
#             cv2.imshow('image', batch_features[i])
#             cv2.waitKey(0)
#             cv2.destroyAllWindows()
             # print(features[index])

         print(batch_features.shape)
         yield batch_features, keras.utils.to_categorical(batch_labels, num_classes)
0 Answers
Related