I have a TensorFlow input pipeline that accepts .npz files which each contain an example. In groups of 5, I am writing them to TFRecords. This is slow. The slowness is np.load() of the npzs which is roughly 10 seconds for each ~50MB file.
Since I will have many .npzs, I'd like to process them in parallel. I'm assuming that np.load is I/O bound. Can I use async/await syntax to make the loading faster? (tf.io.TFRecordWriter seems to be fast enough to ignore). Or would you suggest another approach?