I have a tf.data.Dataset that contains serialized tfrecord/protobuf examples:
>>> example = ds.make_one_shot_iterator().get_next()
>>> example
<tf.Tensor: id=42, shape=(), dtype=string, numpy=b'\n\xdd:\n\r\n\x01y\x12\x08\x12\x06\n\x...'>
So each example in the dataset ds is just a flat byte string.
What I'd like to do is to write these byte strings to disk (tfrecords file). The way I know to do this is to go through python land. For instance, using a tf1 session, this would look like:
example = ds.make_one_shot_iterator().get_next()
with tf.io.TFRecordWriter("/path/to/output/data.tfrecords") as w:
with tf.Session() as sess:
w.write(sess.run(example))
This doesn't feel very efficient to me. So my question is:
- Is there a way that I can write this to a file without having to go through the python layer?