keras MLP halts mid-training without killing the run

user_18923337 0
Viewed 57

I am using optuna v.2.10.0 on a keras v.2.8.0 MLP using Python v.3.9.12 for macOS (tensorflow-metal v.0.4.0), using GPU and at a random point during training of a trial the progress just stops, the GPU usage drops to nothing, but the program doesn't end. I don't know what is causing this. n_trials and timeout are set at a considerable value.

Final lines visible in the run console:

Epoch 103/350
208/208 [==============================] - 2s 10ms/step - loss: 2.3030 - accuracy: 0.1017 - val_loss: 2.3030 - val_accuracy: 0.0978
Epoch 104/350
208/208 [==============================] - 2s 9ms/step - loss: 2.3030 - accuracy: 0.1018 - val_loss: 2.3030 - val_accuracy: 0.0978
Epoch 105/350
 56/208 [=======>......................] - ETA: 1s - loss: 2.3030 - accuracy: 0.1027

This has always happened in the first trial.

Update:

This also happens when not using optuna. The process just stops in the middle of an epoch, as if it is waiting for something and remains like that.

optuna keras python
1 Answers

Hmm... Doesn't the hang happen without Optuna with the same hyperparameter configuration? I suspect that a malicious parameter configuration (e.g. model is too large) is sampled by Optuna.

user_68335230
Related