TF record corrupted after several successful training epochs

user_3568055 0
Viewed 1848

I was training a neural network and had run over all the training data for several epochs successfully. However, the tfrecord corrputed error suddenly came out as follows:

File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/lib/io/tf_record.py", line 77, in tf_record_iterator
  reader.GetNext(status)
File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
  self.gen.next()
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/errors_impl.py", line 466, in raise_exception_on_not_ok_status
  pywrap_tensorflow.TF_GetCode(status))
tensorflow.python.framework.errors_impl.DataLossError: corrupted record at 106241330

I checked the data file again and it was indeed corrupted at that line. But the data was intact before I ran the training code and I simply just read the data by following code:

batch_data = []
record_iterator = tf.python_io.tf_record_iterator(path=file, options=options)
for string_record in record_iterator:
  example = tf.train.Example()
  example.ParseFromString(string_record)
  data = generate_data_from_record(example) # record parsing code
  batch_data.append(data)
  if len(batch_data) == batch_size:
    yield batch_data
    batch_data = []

I am wondering why the data file was corrupted and how can I remain the integrity of the data file.

tfrecord tensorflow
2 Answers

You should make a clean copy of your tfrecord files. Whenever your working copy get corrupted, replace from the clean copy. The dataLoss error seems to be as a result of several reading of the same record, and its also dependent on the disk.

user_96749190

If someone is facing this problem, the above answer by @nwoye-cid worked for me plus the link below to install everything properly.
Also, restart your kernel from scratch if nothing works then only go for other solutions.
Link!

user_117455740
Related