We have several jobs (who have no code in common except Spark lib) that writes parquet snappy files. Randomly, some files are corrupted. The job writing those files doesn't failed and doesn't throw specific Errors on it.
Most of the times, the job write files not corrupted, but sometimes it writes corrupted files and the job doesn't fail. So we only know that files are corrupted when we try to read them.
Thus: Not possible to reproduce easily (very random and rare) and no errors linked to corruption in the job that write the files.
When we try to read these files, we have:
- With Parquet Tools:
OSError: IOError: Corrupt snappy compressed data.
- Spark:
Caused by: java.io.IOException: could not read page Page [bytes.size=1048612, valueCount=58058, uncompressedSize=1048612] in col [client_id] optional binary client_id (UTF8) at org.apache.spark.sql.execution.datasources.parquet.VectorizedColumnReader.readPageV1(VectorizedColumnReader.java:618) at org.apache.spark.sql.execution.datasources.parquet.VectorizedColumnReader.access$000(VectorizedColumnReader.java:49) at org.apache.spark.sql.execution.datasources.parquet.VectorizedColumnReader$1.visit(VectorizedColumnReader.java:547) ... 44 more Caused by: java.io.IOException: FAILED_TO_UNCOMPRESS(5)
- With Athena:
GENERIC_INTERNAL_ERROR: Malformed input: offset=259867
The jobs have worked fine all last year, and we have these since these errors since the 5th of November, since then we haven't redelivered our EMR cluster and code of the jobs.
Our stack:
emr 5.31.0
Spark 2.4.6
openjdk 1.8.0_302
EMRFS consistent view:Disabled
I know it's like a message in the bottle, but we are tuck, and if anyone has encountered the issue and has an idea please don't hesitate, thanks