I am working on a CNN in Google Colab. I have 25 GB of RAM provided by Colab, yet my sheet keeps crushing when I want to load in my input data. My input data are 1800 pictures of size 1080x1920x3. From other questions in this forum I have the suspicion that my code uses too much memory. However, I consider my code already as pretty neat.
Do you know why my Colab keeps crushing and how I can avoid this error?
X = []
y = []
classes = ['asphalt_track_forest', 'asphalt_track_field']
base_path = '/data/img/'
for i, target in enumerate(classes):
files = os.listdir(base_path+target)
for file in files:
# load the image
img = load_img(base_path+target+'/'+file)
img = np.int32(img)
# convert it to an array
img_array = img_to_array(img)
# append the array to X
X.append(img_array)
# append the numeric target to y
y.append(i)
X = np.array(X)
y = np.array(y)
shuffler = np.random.permutation(len(X))
X = X[shuffler]
y = y[shuffler]
print(f'X input:{X.shape}')
print(f'y input:{y.shape}')