How to decode examples in Tensorflow 2 (porting from 1.12)

user_826983 0
Viewed 205

I have the following method which is supposed to decode samples from serialized TFRecordDataset:

def decode_example(self, serialized_example):
    """Return a dict of Tensors from a serialized tensorflow.Example."""
    data_fields, data_items_to_decoders = self.example_reading_spec()
    # Necessary to rejoin examples in the correct order with the Cloud ML Engine
    # batch prediction API.
    data_fields['batch_prediction_key'] = tf.io.FixedLenFeature([1], tf.int64, 0)
    if data_items_to_decoders is None:
        data_items_to_decoders = {
            field: tf.contrib.slim.tfexample_decoder.Tensor(field)
            for field in data_fields
        }

    decoder = tf.contrib.slim.tfexample_decoder.TFExampleDecoder(data_fields, data_items_to_decoders)

    decode_items = list(sorted(data_items_to_decoders))
    decoded = decoder.decode(serialized_example, items=decode_items)
    return dict(zip(decode_items, decoded))

However, this does not work under Tensorflow 2.

tf.contrib does not exist anymore and I do not find anything I can use to decode these examples.

I cannot even find the TFExampleDecoder after installing tensorflow-data-validation.

Any idea what's wrong there and/or how I can decode my examples?

tensorflow2.0 tensorflow python
1 Answers

I was able to make it work using tf.io.parse_single_example.

We have to declare our data fields as usual (example_reading_spec) which we then can use to decode an example:

def example_reading_spec():

    data_fields = {
        'inputs': tf.io.VarLenFeature(tf.float32),
        'targets': tf.io.VarLenFeature(tf.int64),
    }

    return data_fields

def decode_example(serialized_example):
    """Return a dict of Tensors from a serialized tensorflow.Example."""
    return tf.io.parse_single_example(
        serialized_example,
        features=example_reading_spec()
    )

Now we can use Dataset.map to load our dataset shards like this:

record_dataset = tf.data.TFRecordDataset(filenames, buffer_size=1024)
record_dataset = record_dataset.map(decode_example)
user_8269830
Related