I was trying to train a multi-head attention model on some open angle data that I had generated, for a classification problem. The model was meant to test for testing what normalizations to use for my data, which can be found here. A colab notebook with the relevant code can also be found here.
I was trying to train the model over 90 epochs but each time after somewhere between 15 and 25 epochs the model crashes out with no error message.
Here is a snippet of the output
Epoch 14/90
491/491 [==============================] - 60s 122ms/step - loss: 0.9339 - sparse_categorical_accuracy: 0.5995 - val_loss: 0.9894 - val_sparse_categorical_accuracy: 0.5867
Epoch 15/90
491/491 [==============================] - 60s 122ms/step - loss: 0.9312 - sparse_categorical_accuracy: 0.5992 - val_loss: 0.9819 - val_sparse_categorical_accuracy: 0.5898
116/116 [==============================] - 6s 54ms/step - loss: 0.9647 - sparse_categorical_accuracy: 0.5941
I have tried running the code on a Google Compute Engine GPU through colab as well as on my local system in a wsl2 environment, each attempt giving me the same results. The same problem occurs even if I don't normalize the data.
Tech Specs: Windows 11 Home AMD ryzen 7 5800H Nvidia RTX 3050 mobile 16GB RAM VSCode on WSL2 Google Colab with a Python 3 Google Compute Engine backend (GPU)