Create unsupervised embedding model in keras?

Question

I want to create an autoencoder with the following architecture:

    path_source_token_input = Input(shape=(MAX_CONTEXTS,), dtype=tf.int32, name='source_token_input')
    path_input = Input(shape=(MAX_CONTEXTS,), dtype=tf.int32, name='path_input')
    path_target_token_input = Input(shape=(MAX_CONTEXTS,), dtype=tf.int32, name='target_token_input')

    paths_embedded = Embedding(PATH_SIZE, DEFAULT_EMBEDDINGS_SIZE, name='path_embedding')(path_input)

    token_embedding_shared_layer = Embedding(TOKEN_SIZE, DEFAULT_EMBEDDINGS_SIZE, name='token_embedding')
    path_source_token_embedded = token_embedding_shared_layer(path_source_token_input)
    path_target_token_embedded = token_embedding_shared_layer(path_target_token_input)

    context_embedded = Concatenate()([path_source_token_embedded, paths_embedded, path_target_token_embedded])  # --> this up to now, is the output of the STANDALONE embedding model

    -------- SPLIT HERE? ------

    context_after_dense = TimeDistributed(Dense(CODE_VECTOR_SIZE, use_bias=False, activation='tanh'))(context_embedded)  # in short, this layer probably has to stay

    encoded = LSTM(100, activation='relu', input_shape=context_after_dense.shape)(context_after_dense)
    decoded = RepeatVector(MAX_CONTEXTS)(encoded)
    decoded = LSTM(100, activation='relu', return_sequences=True)(decoded)
    result = TimeDistributed(Dense(1), name='PROBLEM_is_here')(decoded)  # this seems to be some trick according to https://github.com/keras-team/keras/issues/10753, so probably don't remove

    inputs = (path_source_token_input, path_input, path_target_token_input)
    model = tf.keras.Model(inputs=inputs, outputs=result)

So far I have learned that it is impossible to implement an inversed embedding layer in the decoder, so naturally my conclusion is to split my network in two: one to generate concatenated embeddings of input, and the second part would be the autoencoder itself with input as concatenated embeddings (output of first part). Now my question is, is it possible to create an unsupervised embedding model in keras? or anywhere else for that matter. My data is unlabeled and the point of my final neural network would be to create clusters of said unlabeled data.

Create unsupervised embedding model in keras?

0 Answers