How to write a dataset of serialized examples directly to a tfrecords file?

Viewed 243

I have a tf.data.Dataset that contains serialized tfrecord/protobuf examples:

>>> example = ds.make_one_shot_iterator().get_next()
>>> example
<tf.Tensor: id=42, shape=(), dtype=string, numpy=b'\n\xdd:\n\r\n\x01y\x12\x08\x12\x06\n\x...'>

So each example in the dataset ds is just a flat byte string.

What I'd like to do is to write these byte strings to disk (tfrecords file). The way I know to do this is to go through python land. For instance, using a tf1 session, this would look like:

example = ds.make_one_shot_iterator().get_next()

with tf.io.TFRecordWriter("/path/to/output/data.tfrecords") as w:
    with tf.Session() as sess:
        w.write(sess.run(example))

This doesn't feel very efficient to me. So my question is:

  • Is there a way that I can write this to a file without having to go through the python layer?
0 Answers
Related