How to handle incorrect parquet schema of some files while reading from HDFS directory?

user_8832641 0
Viewed 290

I am having the below directory in HDFS.

/HDFS/file/date=20200801/id=1
..
/HDFS/file/date=20200831/id=1
/HDFS/file/date=20200901/id=1
/HDFS/file/date=20200902/id=1
/HDFS/file/date=20200903/id=1
...
/HDFS/file/date=20200930/id=1

I am reading these files using

df=spark.read.parquet('/HDFS/file/').option("mergeSchema","true")

The problem is couple of columns in the above file has double format in some files while the same columns have String format in some files.

The merge schema gives me a error that cannot merge schema for double and string type.

Is there a way to handle the schema while reading the parquet files to convert the problem columns to string while reading?

apache-spark-sql apache-spark
1 Answers

I think the best bet would be to describe the schema explicitly and use it to load the partially-incorrect dataset. The trick I'd use would be to choose the "widest" (the most forgiving) data type (e.g. string) and use it for the columns that are affected by this data incorrectness. Once the dataset is loaded, you could then have another pass over it to convert it to the expected type (e.g. double).

Don't know how to define a schema for 10+ fields in Python without much typing as I'm with Scala (that gives case classes that work well with Encoders utility).

user_13053440
Related