Keras model OOM killed during training

Viewed 1339

I am trying to training a Keras seq2seq encoder-decoder model like this:


Layer (type)                    Output Shape         Param #     Connected to                     
==================================================================================================
enc_inputs (InputLayer)         (None, None)         0                                            
__________________________________________________________________________________________________
dec_inputs (InputLayer)         (None, None)         0                                            
__________________________________________________________________________________________________
enc_embedding (Embedding)       (None, None, 256)    1840384     enc_inputs[0][0]                 
__________________________________________________________________________________________________
dec_embedding (Embedding)       (None, None, 256)    1291008     dec_inputs[0][0]                 
__________________________________________________________________________________________________
encoder_lstm (LSTM)             [(None, 256), (None, 525312      enc_embedding[0][0]              
__________________________________________________________________________________________________
dec_lstm (LSTM)                 [(None, None, 256),  525312      dec_embedding[0][0]              
                                                                 encoder_lstm[0][1]               
                                                                 encoder_lstm[0][2]               
__________________________________________________________________________________________________
time_distributed_1 (TimeDistrib (None, None, 5043)   1296051     dec_lstm[0][0]                   
==================================================================================================
Total params: 5,478,067
Trainable params: 5,478,067
Non-trainable params: 0
__________________________________________________________________________________________________
None
Fit model

But it gets killed before any training actually happens:

Process finished with exit code 137 (interrupted by signal 9: SIGKILL)

I've tried reducing the batch size form 64 down to 16 but get the same error.

I'm running from PyCharm on macOS with a python3.6 environment in Conda. My python3.6 process is using lots of virtual RAM and seems to reach about 100GB before being killed. Strangely this used to work with my previous model with about 14m total parameters, but even that model doesn't train any more. I haven't consciously changed anything - have some TF version, some python/Conda environment.

I think the only difference is switching from PyCharm CE to Professional.

So questions:

  1. Can I allow Python to use more virtual memory before being killed?
  2. Is this a config problem with PyCharm Professional?
  3. Can I somehow change my model to require less memory but still train on the same data set?
1 Answers

Looks like a problem with tf.__version == 1.11. I did pip install --upgrade tensorflow==1.10 and the model ran fine.

Related