I am trying to train a neural network with a dataset which have more thant 250k images. But I am stuck because of my limited computer which have 16 GB RAM and 32 GB SWAP. Before I load all the images, it gives me an memory error. So I was wondering if there is a way to train my neural network using all the images I have ? For example, instead of using RAM memory to load images in numpy array, can we load it on the free space disk ?
EDIT 1:
def get_array_image(file, path):
return cv2.imread(path+file)
def generator(features, labels, num_classes, batch_size, path=''):
# Create empty arrays to contain batch of features and labels#
batch_features = np.zeros((batch_size, 28, 28, 3))
batch_labels = np.zeros((batch_size, 1))
while True:
for cpt in range(0, len(features), batch_size):
for i in range(0, batch_size):
index = cpt + i
#print('images : ', index)
batch_features[i] = get_array_image(features[index], path)
batch_labels[i] = labels[index]
yield batch_features, keras.utils.to_categorical(batch_labels, num_classes)
This is my generator I used with fit_generator function. But I have a problem of accuracy. In fact, I tried it on the mnist dataset with a smal Neural Network. If I use the fit function loading all the images (about 60k images for training and 60k for the test) I have about 0.68 of accuracy after one epoch. But with the fit_generator I obtain only 0.1 . Am I doing something wrong with my generator ? When I print the index variable, it seems good.
EDIT 2: I solved my problem but i do not understand why it works. In fact when I create the array outside the loop I obtain low accuracy but when it is inside the accuracy is good with fit_generator. Does someone know what I am missing ?
def generator(features, labels, num_classes, batch_size, path='', dtype=np.uint8):
# Create empty arrays to contain batch of features and labels#
# batch_features = np.ndarray(shape=(batch_size, 28, 28, 3), dtype=dtype)
# batch_labels = np.ndarray(shape=(batch_size, 1), dtype=dtype)
while True:
for cpt in range(0, len(features), batch_size):
batch_features = np.ndarray(shape=(batch_size, 28, 28, 3), dtype=dtype)
batch_labels = np.ndarray(shape=(batch_size, 1), dtype=dtype)
for i in range(0, batch_size):
# index= random.randint(0, len(features)-1)
index = cpt + i
#print('images : ', index)
batch_features[i] = get_array_image(features[index], path)
batch_labels[i] = labels[index]
# print(batch_labels[i])
# cv2.imshow('image', batch_features[i])
# cv2.waitKey(0)
# cv2.destroyAllWindows()
# print(features[index])
print(batch_features.shape)
yield batch_features, keras.utils.to_categorical(batch_labels, num_classes)