I have a pretty simple FFNN in Keras.
model = Sequential()
shared_embedding = Embedding(vocab_size, word_emb_dim, input_length=1, name='embedding')
model.add(shared_embedding)
model.add(Dropout(emb_dropout))
model.add(Dense(hidden_size, name='hidden',
kernel_constraint=max_norm(max_norm_size), activation='relu'))
model.add(shared_embedding)
model.add(Dense(vocab_size, name='output',
kernel_constraint=max_norm(max_norm_size), activation='softmax'))
model.compile(optimizer=optimizer, loss='sparse_categorical_crossentropy')
It's a very, very simple word model that merely predicts bigrams. So given a word in the data, the model is supposed to predict the next word.
I have a vocabulary size of 50,000, and I have about 11 million bigrams. So it goes without saying that I can't store all of the targets in the full one hot encoded matrix.
What I want to do is be able to do is use the integer for the target directly into keras, and force keras to deal with it. I'd have thought specifying the loss as sparse_categorical_crossentropy would've been enough, but I guess not.
Anyone know what to do?