How to build a one hot encoder from a vector of words in TF

user_947659 0
Viewed 37

I have data where each row in the batch is a fixed sized vector of mutually exclusive "words". Ex:

batch:
[1, 2, 3]
[4, 5, 6]
[4, 7, 8]


global dictionary:
{
 0: {1, 4}
 1: {2, 5, 7}
 2: {3, 6, 8}
}

In the above example, for cols 0, 1, and 2...we have vocabs of [1, 4], [2, 5, 7], and [3, 6, 8] which are mutually exclusive. I also have a giant dictionary of all vocab words for all the columns. How do I use that dictionary of dict(column_idx) -> {vocab set} to build a one hot encoder in tensorflow?

For the above example I would want an output of:

[1, 0,    1, 0, 0,   1, 0, 0]
[0, 1,    0, 1, 0,   0, 1, 0]
[0, 1,    0, 0, 1,   0, 0, 1]

with the one hot mappings being:

[1, 4,    2, 5, 7,   3, 6, 8]
tensorflow2.0 tensorflow
1 Answers

The tricky part is that each column needs to be encoded differently. If you break the problem down to encoding one column at a time then it is mostly about deconstructing, encoding and recombining the batched input. Here is a working example in numpy:

import numpy as np

batch = np.array([
[1, 2, 3],
[4, 5, 6],
[4, 7, 8]])


vocab = {
 0: {1, 4},
 1: {2, 5, 7},
 2: {3, 6, 8}
}

# Construct a one hot encoding matrix for each vocabulary
# Here we are using identity matrix as the getter
vocab_eye = {k: np.eye(len(v)) for k, v in vocab.items()}

# Construct a converter to indices, a map from value to index if you will
vocab_map = {k:np.vectorize(list(v).index) for k, v in vocab.items()}

# Deconstruct, encode and merge each column
encoded_cols = [vocab_eye[i][vocab_map[i](col[:,0])]
                for i, col in enumerate(np.split(batch, batch.shape[1], axis=1))]

encoded_batch = np.concatenate(encoded_cols, axis=1)
# Outputs
# array([[1., 0., 1., 0., 0., 0., 1., 0.],
#        [0., 1., 0., 1., 0., 0., 0., 1.],
#        [0., 1., 0., 0., 1., 1., 0., 0.]])

You can perhaps optimise the encoding by already giving split data. So instead of batching everything and splitting, just give 3 separate inputs to the data pipeline in Tensorflow. Then you can imagine having 3 encoding paths one for each feature set, which then gets merged.

user_97589220
Related