I have a fMRI dataset in which some participants watch images and we have the corresponding brain activities.
If an image was "jungle", the corresponding label is either the word2vec representation of "jungle", or cnn features from trained alexnet model (for example, from layer 8).
I am training an autoencoder, in a way that the input is labels (word2vec or cnn features) and I have defined the loss in a way that the hidden layer features also becomes similar to brain features.
When I train the model with cnn features, the models seems to be able to train:
However, when I train the model with word2vec network, it fails:
I have 1200 samples for training.
Is it because of the nature of word2vec embeddings or should I play with the hyperparameters. If I have to tune the hyperparameters, how should I fix them when the training curve is increasing?