GPU Memory is full directly after sequential model initialization and wont change during training

user_5157887 0
Viewed 714

I am using Keras, on the backend tensorflow on windows 7 with the NVIDIA Quadro M2000M GPU.

When i initialization my model which contains 5 GRU, 5 Dropout and 1 Dense layers the GPU memory usage jumps to 3800MB of 4096MB and stays there until i restart my spyder session. Clearing the session within spyder with:

K.clear_session()

does not work. The Memory usage stays at that high level.

Is it normal that such a model allocate this much memory of the GPU? What can i change so the memory usage can be used proberly? I want to improve the training speed and i think this high memory usage hinder the GPU to use her full potential.

Update

My model looks like this:

model = Sequential()
layers = [1, 70, 50,100, 50,20, 1]

model.add(GRU(
        layers[1],
        #batch_size = 32,
        input_shape=(sequence_length, anzahl_features),
        return_sequences=True))
model.add(Dropout(dropout_1))
model.add(GRU(
        layers[2],
        #batch_size = 32,
        return_sequences=True))
model.add(Dropout(dropout_2))
model.add(GRU(
        layers[3],
        #batch_size = 32,
        return_sequences=True))
model.add(Dropout(dropout_3))
model.add(GRU(
        layers[4],
        #batch_size = 32,
        return_sequences=True))
model.add(Dropout(dropout_4))
model.add(GRU(
        layers[5],
        #batch_size = 32,
        return_sequences=False))
model.add(Dropout(dropout_5))
model.add(Dense(
         layers[6]))
model.add(Activation('sigmoid'))

My feature matrix has the size 506x500x35 (506 examples, 500 sequence length and 35 features). The batch size is set to 128. Site note: I am not saying that that is the perfect feature matrix or model configuration.

Here also a screenshot of GPU-Z where i restarted spyder and started the model until the second epoch:

enter image description here

Model Summary

keras tensorflow spyder memory gpu
2 Answers

Let me just give the full piece of code how to control memory usage on your GPU:

    from keras.backend.tensorflow_backend 
    import set_session config = tf.ConfigProto() 

    config.gpu_options.per_process_gpu_memory_fraction = 0.9
    config.gpu_options.allow_growth = True set_session(tf.Session(config=config))

GPU speed training might depends on your batch size, is it the same on CPU and GPU. Is it cuda installed

user_17393250
Related