Tensorflow Keras Neural Network Model Clear GPU Memory

Question

I am trying to build a neural network with keras. I am using Windows 10 and installed tensorflow on my GPU NVIDA GeForce 1060 therefore I am using CUDA : 10.0 and tensorflow 2.3.0.

Basically this works if I run it one time alone but I now want to find the best configuration for this network therefore I am trying to build the model every time and then find the best. I am planning on doing that in an for loop.
But my model depends on exactly that parameters data, the model look like:

model_heightmap = Sequential(
        [
            # Input(shape=(140, 150)),
            # Reshape(target_shape=(140 * 150,), input_shape=(140, 150)),
            Flatten(input_shape=(140, 150)),
            Dense(8000, activation='relu'),
            Dense(5000, activation='relu'),
            Dense(3000, activation='relu'),
            Dense(500, activation='relu'),
            Dense(200, activation='relu'),
            Dense(5, activation='softmax'),
            Dense(df_max_cat + 1, activation='softmax')
        ]
    )

So here is the df_max_cat the changing factor.
But when the for loop comes to its second iteration it gets the following error: 2020-10-03 13:45:29.899409: I tensorflow/stream_executor/cuda/cuda_driver.cc:775] failed to allocate 818.20M (857944832 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
I know that this mistakes happens because I run out of memory and this happens because I create the model every time new.
I don't need the old models so how am I able to clear up the old GPU Memory Space for the next run ???
I have already tried:

del model_heightmap  # for avoid any trace on aigen
tf.keras.backend.clear_session()  # removing session, it will instance another

Does anybody has any ideas how to solve this problem ?

Tensorflow Keras Neural Network Model Clear GPU Memory

0 Answers