I am having some problems to understand the functioning of a Dense layer handling text sequences. Let's imagine this simple case: I have two sentences and I assign integers to the words:
Sentence 1 I like cheese
Sentence 2 I like milk
Sentence to sequence
Seq1 [1, 2, 3]
Seq2 [1, 2, 4]
Then we take the input (the integer sequences) and add them to a Embedding layer (random numbers in 2 dimensions):
Embedding matrix
Term Index Vector
I 1 [0.2 0.6]
like 2 [0.7 0.1]
cheese 3 [0.4 0.5]
milk 4 [0.1 0.9]
Next step is to Flatten the sequences with the embeddings in order to make it 1D:
Sequence to Embedding
Seq1 [[0.2 0.6] [0.7 0.1] [0.4 0.5]]
Seq2 [[0.2 0.6] [0.7 0.1] [0.1 0.9]]
Flatten to 1D for Dense layer
Seq1 [0.2 0.6 0.7 0.1 0.4 0.5]
Seq2 [0.2 0.6 0.7 0.1 0.1 0.9]
Now we can use those arrays as input for the Dense layer, something that would look like this:
Dense layer of 3 units with input length of sequence. Dot product of the input sequence with the weights matrix of the dense layer.
Dense layer
Seq1 X1 X2 X3 X4 X5 X6 · WH = ZH …
0.2 0.6 0.7 0.1 0.4 0.5 · W1,1 W1,2 W1,3 +
W2,1 W2,2 W2,3
W3,1 W3,2 W3,3 BH
W4,1 W4,2 W4,3
W5,1 W5,2 W5,3
W6,1 W6,2 W6,3
Seq2 X1 X2 X3 X4 X5 X6 · WH = ZH …
0.2 0.6 0.7 0.1 0.1 0.9 · W1,1 W1,2 W1,3 +
W2,1 W2,2 W2,3
W3,1 W3,2 W3,3 BH
W4,1 W4,2 W4,3
W5,1 W5,2 W5,3
W6,1 W6,2 W6,3
The problem here is that every feature corresponds to one part of the sequence, X5 and X6 belong to the vector of the last word. Normally, with Bag of Words input every X is assigned to a word, and if the word is present then it gets a weight (following TF-IDF for example). Also with numerical data, Xi represents a feature (price, temperature, GDP...) and it is always that case. Here, however, X is not assigned to a word and depends on the order. See X5 and X6 how they change because the words are different.
The weights in the Weight Matrix of the Dense layer are assigned to a feature (Xi), and then they are optimised.
My question:
How does it work if the order of words changes all the time and Xi is referring to different words?
I understand LSTMs and other recurrent networks can handle dynamic ordering, but Dense layers seemed to me that could not work with sequential text and that the input should be fixed by One Hot vector or TF-IDF for example. Still I have seen examples of models with Sentences to Sequences of Integers, Embedding, Flatten and Dense layer architectures, plus I have tried myself and I see it does work... I would really appreciate some explanation or correction in my flow of thinking. Thanks!