How to debug out of memory in tensorflow2-gpu

user_7381947 0
Viewed 515

I am using tensorflow-2 gpu with tf.data.Dataset. Training on small models works.

When training a bigger model, everything works at first : gpu is used, the first epoch works with no trouble (but I am using most of my gpu memory).

At validation time, I run into a CUDA_ERROR_OUT_OF_MEMORY with various allocation with a smaller and smaller amount of bytes that could not be allocated (ranging from 922Mb to 337Mb). I currently have no metrics and no callbacks and am using tf.keras.Model.fit. If I remove the validation data, the training continues.

What is my issue ? how can I debug this ?

In tf1, I could use RunOptions(report_tensor_allocations_upon_oom=True), does any equivalent exist in tf2 ?

This occurs with tensorflow==2.1.0 .

tensorflow2.x gpu python
1 Answers

These did not occur in 2.0 alpha TensorFlow but in 2.0.

Pip installs tensorflow-gpu==2.0.0: has leaked memory!
Pip install tensorflow-gpu==2.0.0-alpha: it's all right!

Try it out

user_67929280
Related