org.apache.spark.sql.AnalysisException: Illegal Parquet type: INT64 (TIMESTAMP(NANOS,true))

user_19325274 0
Viewed 152

I have a table in postgres containing a timestamp column. This table is stored in hdfs as a parquet file. When I try to read it:

parqDF = sc.read.parquet("hdfs://localhost:9000/hadoop_files/sample_2022_02.parquet")

the app fails with the error:

org.apache.spark.sql.AnalysisException: Illegal Parquet type: INT64 (TIMESTAMP(NANOS,true))

Do I need to change the column type in db to something else? Or add some conversion in the app?

parquet apache-spark postgresql
1 Answers

The type INT64 is not supported in the old versions of spark, try to load the file using spark 3.0.0+ (you can try with spark 2.4.x).

user_95605940
Related