How to write LabelEncoder in Tensorflow?

user_5613170 0
Viewed 2801

I'm trying to parse directories on Google Storage into strings, but I keep getting errors. I want to find the directory of each file and return a numerical encoding of the directory name as a Dataset. This would be trivial in sklearn using LabelEncoder, but I'm having trouble doing this in Tensorflow.

CLASS_NAMES = [b'class_1', b'class_2', b'class_3']
labeler = tfds.features.ClassLabel(names=CLASS_NAMES)

def parse_filenames(filename):
    label = tf.strings.split(tf.expand_dims(filename, axis=-1), sep='/')
    label = label.values[-2]

    # Problem is in the two lines below
    position_feature = tf.feature_column.categorical_column_with_vocabulary_list('label_names', CLASS_NAMES)
    label = tf.io.parse_example(label, features=position_feature)

    return label

folder = b'gs://<bucket>/train/*/*.jpg'
filenames_dataset = tf.data.Dataset.list_files(folder)
label_dataset = filenames_dataset.map(parse_filenames)

next(iter(label_dataset))

I get an error ValueError: dictionary update sequence element #0 has length 16; 2 is required

If I take out the two lines under the "# Problem is here" comment, it works fine, except it returns a string instead of an integer. I've tried other non-tensorflow options, such as <list_name>.index(label), but those of course fail because everything is a tensor instead of a string. Is there another way to do this?

keras tensorflow python
2 Answers

Maybe you can try this line instead of these two lines:

label = tf.argmax(tf.cast(parts[-2] == CLASS_NAMES, tf.int32))

You will get something like [0, 1, 0] (the index of the label in CLASS_NAMES).

Functioning and reproducible example:

import tensorflow as tf
import numpy as np
from string import ascii_lowercase as letters

CLASS_NAMES = [b'class_1', b'class_2', b'class_3']

files = ['\\'.join([np.random.choice(CLASS_NAMES).decode(),
                    ''.join(np.random.choice(list(letters), 5)) + '.jpg']) 
         for i in range(10)]

ds = tf.data.Dataset.from_tensor_slices(files)

Here are the fake files I generated:

['class_3\\jrxog.jpg',
 'class_1\\slfiq.jpg',
 'class_2\\svldd.jpg',
 'class_2\\avrgt.jpg',
 'class_3\\wqwuv.jpg']

Now implement this:

def get_label(file_path):
    parts = tf.strings.split(file_path, '\\')
    return file_path, tf.argmax(tf.cast(parts[-2] == CLASS_NAMES, tf.int32))

ds = ds.map(get_label)

next(iter(ds))

(<tf.Tensor: shape=(), dtype=string, numpy=b'class_1\\bbqrx.jpg'>,
 <tf.Tensor: shape=(), dtype=int64, numpy=0>)
user_109083750

i used the label encoder from sklearn. This is something that you may have to pickle, that way you can reverse transform your outcome later on. Im still new at this so im not sure how well this might work for you

user_140737260
Related