I am trying to query a 50 gb data from athena, I have 3 similar files and one of those file is throwing an error while the other two files are working properly.
HIVE_BAD_DATA: Line too long in text file: s3://my-bucket/my-file.txt
What might have caused the problem? Is there any way I can check the issue using SQL or python?
It looks like one of the row/column exceeds the limit set by Athena which is 32 MB. If the other two files are working then you need to verify which row is larger in length and fix it.
Below is more about this limitation in Athena :
Row or column size limitation – The size of a single row or its columns cannot exceed 32 megabytes. This limit can be exceeded when, for example, a row in a CSV or JSON file contains a single column of 100 megabytes. Exceeding this limit can also produce the error message Line too long in text file. To work around this limitation, make sure that the sum of the data of the columns in any row is less than 32MB.
Ref https://docs.aws.amazon.com/athena/latest/ug/other-notable-limitations.html