How to use TensorBoard with Keras in Python for visualizing embeddings

I'm also reading the book "Deep Learning with Python" which uses Keras. Here is my solution to this question. First，I try this code:

callbacks = [keras.callbacks.TensorBoard(
    log_dir = 'my_log_dir',
    histogram_freq = 1,
    embeddings_freq = 1,
    embeddings_data = x_train,
)]
history = model.fit(x_train, y_train, epochs=2, batch_size=128, validation_split=0.2, callbacks=callbacks)

But there is an error: ResourceExhaustedError.

Because there are 25000 samples in "x_train", it is hard to embedding all of them on my old notebook. So next I try to embedding the first 100 samples of "x_train", and it makes sense.

The code and result are showed here.

callbacks = [keras.callbacks.TensorBoard(
    log_dir = 'my_log_dir',
    histogram_freq = 1,
    embeddings_freq = 1,
    embeddings_data = x_train[:100],
)]
history = model.fit(x_train, y_train, epochs=2, batch_size=128, validation_split=0.2, callbacks=callbacks)

Projector of 100 samples

Note that in the projector, "Points: 100" means there are 100 samples, and "Dimension: 64000" means the embedding vector length for one sample is 64000. There are 500 words in one sample, as "max_len = 500", and there is a 128_dim vector for each word, so 500 * 128 = 64000.

Yes that is correct, you need to provide what to embed for the visualisation using the embeddings_data argument:

callbacks = [
    keras.callbacks.TensorBoard(
        log_dir='my_log_dir',
        histogram_freq=1,
        embeddings_freq=1,
        embeddings_data=np.array([3,4,2,5,2,...]),
    )
]

embeddings_data: data to be embedded at layers specified in embeddings_layer_names. Numpy array (if the model has a single input) or list of Numpy arrays (if the model has multiple inputs).

Have a look at the documentation for updated information on what those arguments are.

I just asked myself the same question. From the documentation it wasn't that clear what "embeddings_data" was supposed to be. But it makes sense that it's a list of the words to embed, or rather their indices.

But I am a bit confused about the format. From my tokenization with the keras tokenizer I have a "word_index" with a number for all the words in my vocabulary from which the most frequent maxwords entries are used. The tokenizer gives me lists of integers from 0 to maxwords or a one-hot-encoded bag of words. I then pad those sequences to maxlen.

max_words = 1000

tokenizer = Tokenizer(num_words=max_words)
tokenizer.fit_on_texts(X_train)

X_train_sequences = tokenizer.texts_to_sequences(X_train)
X_train_one_hot_results = tokenizer.texts_to_matrix(X_train, mode='binary')

X_test_sequences = tokenizer.texts_to_sequences(X_test)
X_test_one_hot_results = tokenizer.texts_to_matrix(X_test, mode='binary')

#####

maxlen = 100
X_train_pad = pad_sequences(X_train_sequences, maxlen=maxlen)
X_test_pad = pad_sequences(X_test_sequences, maxlen=maxlen)

I then create a model that starts with an embedding layer as such:

embedding_dim = 300

model = Sequential()
model.add(Embedding(max_words,embedding_dim,input_length=maxlen, name='embed'))
model.add(Flatten())
...

So max_words is my vocabulary size, the size of a one hot encoded word/text, embedding_dim is the size of the output of the layer and maxlen is the length of the sequence, i.e. the number of words in a sentence, which is kept constant by padding. Are the numbers from the word_index the ones that "embeddings_data" should get somehow?

When I enter "only" a numpy array that lists an arbitrary number of word indices

...
tensorboard = TensorBoard(log_dir='log_dir', histogram_freq=1, embeddings_freq=1, embeddings_data=np.array([1,2,3]))
...

I get this error:

ValueError: Cannot feed value of shape (3, 1) for Tensor 'embed_input_2:0', which has shape '(?, 100)'

In my example 100 is the length of the sequence. That is confusing. Because I want to visualize single words, not sentences or texts, don't I? Apparantly the Callback wants to feed the layer one or more sequences.

So how am I supposed to encode my "Queen", "King", "woman", "man" tokens so I can see the semantic relations (hopefully)? Or more generally: How to get an overview over all vocabulary elements to spot overall trends in individually trained embeddings?

Here is an example using embeddings for a basic MNIST convolutional NN classifier. The embedding_data happens to be the input data in this scenario, and I believe it will typically be whatever data is fed forward through the network.

Here's the linked script with some commentary. First, they start with the basic MNIST setup.

'''Trains a simple convnet on the MNIST dataset and embeds test data.
The test data is embedded using the weights of the final dense layer, just
before the classification head. This embedding can then be visualized using
TensorBoard's Embedding Projector.
'''

from __future__ import print_function

from os import makedirs
from os.path import exists, join

import keras
from keras.callbacks import TensorBoard
from keras.datasets import mnist
from keras.models import Sequential
from keras.layers import Dense, Dropout, Flatten
from keras.layers import Conv2D, MaxPooling2D
from keras import backend as K

import numpy as np

batch_size = 128
num_classes = 10
epochs = 12
log_dir = './logs'

if not exists(log_dir):
    makedirs(log_dir)

# input image dimensions
img_rows, img_cols = 28, 28

# the data, split between train and test sets
(x_train, y_train), (x_test, y_test) = mnist.load_data()

if K.image_data_format() == 'channels_first':
    x_train = x_train.reshape(x_train.shape[0], 1, img_rows, img_cols)
    x_test = x_test.reshape(x_test.shape[0], 1, img_rows, img_cols)
    input_shape = (1, img_rows, img_cols)
else:
    x_train = x_train.reshape(x_train.shape[0], img_rows, img_cols, 1)
    x_test = x_test.reshape(x_test.shape[0], img_rows, img_cols, 1)
    input_shape = (img_rows, img_cols, 1)

x_train = x_train.astype('float32')
x_test = x_test.astype('float32')
x_train /= 255
x_test /= 255
print('x_train shape:', x_train.shape)
print(x_train.shape[0], 'train samples')
print(x_test.shape[0], 'test samples')

Now metadata is created for tensorboard. For this classification task, the embedding_data is the test set. Then the metadata must include the corresponding digit labels.

# save class labels to disk to color data points in TensorBoard accordingly
with open(join(log_dir, 'metadata.tsv'), 'w') as f:
    np.savetxt(f, y_test)

# convert class vectors to binary class matrices
y_train = keras.utils.to_categorical(y_train, num_classes)
y_test = keras.utils.to_categorical(y_test, num_classes)

Now define the tensorboard callback. Note that 'features' will refer to the layer embedding of interest, which is named during model construction. The embedding_data is simply the test set here.

tensorboard = TensorBoard(batch_size=batch_size,
                          embeddings_freq=1,
                          embeddings_layer_names=['features'],
                          embeddings_metadata='metadata.tsv',
                          embeddings_data=x_test)

model = Sequential()
model.add(Conv2D(32, kernel_size=(3, 3),
                 activation='relu',
                 input_shape=input_shape))
model.add(Conv2D(64, (3, 3), activation='relu'))
model.add(MaxPooling2D(pool_size=(2, 2)))
model.add(Dropout(0.25))
model.add(Flatten())

The next layer is named 'features'.

model.add(Dense(128, activation='relu', name='features'))
model.add(Dropout(0.5))
model.add(Dense(num_classes, activation='softmax'))

model.compile(loss=keras.losses.categorical_crossentropy,
              optimizer=keras.optimizers.Adadelta(),
              metrics=['accuracy'])

model.fit(x_train, y_train,
          batch_size=batch_size,
          callbacks=[tensorboard],
          epochs=epochs,
          verbose=1,
          validation_data=(x_test, y_test))
score = model.evaluate(x_test, y_test, verbose=0)
print('Test loss:', score[0])
print('Test accuracy:', score[1])

# You can now launch tensorboard with `tensorboard --logdir=./logs` on your
# command line and then go to http://localhost:6006/#projector to view the
# embeddings