I am trying to training a Keras seq2seq encoder-decoder model like this:
Layer (type) Output Shape Param # Connected to
==================================================================================================
enc_inputs (InputLayer) (None, None) 0
__________________________________________________________________________________________________
dec_inputs (InputLayer) (None, None) 0
__________________________________________________________________________________________________
enc_embedding (Embedding) (None, None, 256) 1840384 enc_inputs[0][0]
__________________________________________________________________________________________________
dec_embedding (Embedding) (None, None, 256) 1291008 dec_inputs[0][0]
__________________________________________________________________________________________________
encoder_lstm (LSTM) [(None, 256), (None, 525312 enc_embedding[0][0]
__________________________________________________________________________________________________
dec_lstm (LSTM) [(None, None, 256), 525312 dec_embedding[0][0]
encoder_lstm[0][1]
encoder_lstm[0][2]
__________________________________________________________________________________________________
time_distributed_1 (TimeDistrib (None, None, 5043) 1296051 dec_lstm[0][0]
==================================================================================================
Total params: 5,478,067
Trainable params: 5,478,067
Non-trainable params: 0
__________________________________________________________________________________________________
None
Fit model
But it gets killed before any training actually happens:
Process finished with exit code 137 (interrupted by signal 9: SIGKILL)
I've tried reducing the batch size form 64 down to 16 but get the same error.
I'm running from PyCharm on macOS with a python3.6 environment in Conda. My python3.6 process is using lots of virtual RAM and seems to reach about 100GB before being killed. Strangely this used to work with my previous model with about 14m total parameters, but even that model doesn't train any more. I haven't consciously changed anything - have some TF version, some python/Conda environment.
I think the only difference is switching from PyCharm CE to Professional.
So questions:
- Can I allow Python to use more virtual memory before being killed?
- Is this a config problem with PyCharm Professional?
- Can I somehow change my model to require less memory but still train on the same data set?