Found array with dim 3. StandardScaler expected <= 2,Unable to allocate 15.5 GiB for an array with shape (34997, 244, 244) and data type float64

Admin 1
Viewed 74

I am trying to normalise the pixel values of all the images contained in a folder at once but the error shows up

def resize():
        data = []
        img_size = 244
        data_dir = r'C:\technocolab project2\archive\img'
        for img in os.listdir(data_dir):
            try:
                imgPath = os.path.join(data_dir,img)
                images = cv2.imread(imgPath, cv2.IMREAD_GRAYSCALE)
                image_resized = cv2.resize(images,(img_size,img_size))
                data.append(image_resized)
            #except Exception as e:
                #print(e)
            except:
                pass
        return data

data = resize()
print(len(data))

sample = data[1]
print(sample.shape)

it prints 244,244

training = data[:int(0.7*len(data))]
validation = data[int(0.7*len(data)):int(0.9*len(data))]
testing = data[int(0.9*len(data)):]

from sklearn.preprocessing import StandardScaler
sc = StandardScaler()
train_normalised = sc.fit_transform(training)
valid_normalised = sc.transform(validation)
test_normalised = sc.transform(testing)

ValueError: Found array with dim 3. StandardScaler expected <= 2.

from sklearn.decomposition import PCA
pca = PCA(n_components = 2)
train_normalised = pca.fit_transform(training)
valid_normalised = pca.transform(validation)
test_normalised = pca.transform(testing)

MemoryError: Unable to allocate 15.5 GiB for an array with shape (34997, 244, 244) and data type float64

norm = sc.fit_transform(image_resized)

NameError: name 'image_resized' is not defined

scikit-learn pca computer-vision image-processing python
1 Answers

First, standard scaler only works for 2D arrays, so you need to reshape your array before calling it.

Second, it is casting your data from np.uint8 to np.float64. Try to do it yourself and make sure everything is in np.float32, which is usually enough.

Another point to consider is loading in memory one dataset at a time (train, validation and test), and load them to a numpy array straightforward instead of creating a list: just create an empty numpy array and start to fill it with the images you read.

Last, but not least, a side note: the way you are splitting your dataset looks weird. Consider that os.listdir does not guarantee any order, so each time you run this code you may get different splits. In addition to this, you should shuffle your data before splitting, otherwise you may be adding some bias to your dataset. Take a look at train_test_split from sklearn.

user_49131430
Related