What do the arguments for TFRecordOptions actually mean (wrt tf.io.TFRecordWriter)?

user_3406189 0
Viewed 232

I export some fairly large Pandas dataframes to Tensorflow's serialized format. And I do it often and it's really slow. Which is probably because I have to serialize the individual examples idk. Also, I compress the files with the "GZIP" option.

I have found some options for the TFRecordWriter in the documentation that look like they might help (buffers help, right?). But there is no explanation of what input_buffer_size does or what range the values might take. Is it {0, 1, 2, 3} or a couple of million? Or do I want output_buffer_size or mem_level or something else?

From the Tensorflow 2.5 documentation:

Args
compression_type        "GZIP", "ZLIB", or "" (no compression).
flush_mode              flush mode or None, Default: Z_NO_FLUSH.
input_buffer_size       int or None.
output_buffer_size      int or None.
window_bits             int or None.
compression_level       0 to 9, or None.
compression_method      compression method or None.
mem_level               1 to 9, or None.
compression_strategy    strategy or None. Default: Z_DEFAULT_STRATEGY.
tfrecord tensorflow python
1 Answers

Some description is given in tf_record.py. That will point to the zlib library which gives some more complete information.

Sadly, the text is in the __init__() method so don't seem to appear in the Tensorflow autodocs

I won't post an "answer" as reading the comments is easy once you find the link and they will remain more accurate than anything I would write.

As a small note, the default for the *_buffer_size is 25,600.

user_21841220
Related