How to use numpy file without importing into RAM?

user_13720565 0
Viewed 1403

I want to use an numpy file (.npy) from Google Drive into Google Colab without importing it into the RAM.

I am working on Image Classification and have my image data into four numpy files in Google Drive. The collective size of the files is greater than 14 GB. Whereas Google Colab only offers 12 GB RAM for usage. Is there a way through which I can use it by loading only single batch at a time into the ram to train the model and removing it from the ram (maybe similar to flow_from_directory)?

The problem using flow_from_directory is that it is very slow even for one block of VGG16 even if I have images in Colab directory.

I am using Cats vs Dogs Classifier dataset from Kaggle.

! kaggle competitions download -c 'dogs-vs-cats'

I converted the image data into numpy array, and saved it in 4 files:

X_train - float32 - 10.62GB - (18941, 224, 224, 3)

X_test - float32 - 3.4GB - (6059, 224, 224, 3)

Y_train - float64 - 148KB - (18941)

Y_test - float64 - 47KB - (6059)

When I run the following code, the session crashes showing 'Your session crashed after using all available RAM.' error.

import numpy as np
X_train = np.load('Cat_Dog_Classifier/X_train.npy')
Y_train = np.load('Cat_Dog_Classifier/Y_train.npy')
X_test = np.load('Cat_Dog_Classifier/X_test.npy')
Y_test = np.load('Cat_Dog_Classifier/Y_test.npy')

Is there any way to use these 4 files without loading it into the RAM?

google-colaboratory numpy python
2 Answers

You can do this by opening your file as a memory-mapped array.

For example:

import sys
import numpy as np

# Create a npy file
x = np.random.rand(1000, 1000)
np.save('mydata.npy', x)

# Load as a normal array
y = np.load('mydata.npy')
sys.getsizeof(y)
# 8000112

# Load as a memory-mapped array
y = np.load('mydata.npy', mmap_mode='r')
sys.getsizeof(y)
# 136

The second array acts like a normal array, but is backed by disk rather than RAM. Be aware that this will cause operations over the arrays to be much slower than normal RAM-backed arrays; often mem-mapping is used to conveniently access portions of the array without having to load the full array into RAM.

user_29378310

The combined size of the files is 14 GB, which is indeed greater than the 12 GB that you say you have available. However, you created those files from data that was in memory as well, as in an earlier version of your question, which suggests that there is enough memory to hold all the data:

save('drive/My Drive/ML/Cats_vs_Dogs_Classifier/X_train.npy', X_train)
save('drive/My Drive/ML/Cats_vs_Dogs_Classifier/Y_train.npy', Y_train)
save('drive/My Drive/ML/Cats_vs_Dogs_Classifier/X_test.npy', X_test)
save('drive/My Drive/ML/Cats_vs_Dogs_Classifier/Y_test.npy', Y_test)

However, if you attempt to load the X_train file again in the same Python session (I assume you're using Jupyter Notebook), you'll temporarily need another 10.6 GB of memory before the 10.6 GB occupied by the previous X_train is released.

You can pick on of the following strategies:

  • Start a new Python process (or kernel) before loading data.
  • Explicitly free the memory before continuing: 
    del X_train, Y_train, X_test, Y_test
  • Put the code that generates the data inside a function. All local variables created in the function will automatically be deleted when the function returns.
user_62288910
Related