Keras ImageDataGenerator Fit causes memory leak

user_1381666 0
Viewed 1159

I'm on Keras 2.2.2 and I'm trying to generate augmentations of my training data with zca_whitening and an ImageDataGenerator. But when I try to fit the generator (which is mandatory when using zca_whitening) the python process eats more and more memory (100Gb+) until it gets killed by the system.

This small example can cause the leak:

import numpy as np
from keras.preprocessing.image import ImageDataGenerator

def cause_leak():
    idg = ImageDataGenerator(zca_whitening = True)
    random_sample = np.random.random((1, 250, 250, 3))
    idg.fit(random_sample)

cause_leak()

Update: Yesterday this was marked as a bug in the Keras repository.

keras tensorflow python
1 Answers

As discussed in this issue this is not a memory leak as computing Singular Value Decomposition on the matrix with (250 * 250 * 3) i.e., 187000 elements is memory intensive. Unfortunately, there is no immediate work around for this issue as of now as problem lies with the calculation of sigma matrix as mentioned here

user_115304620
Related