Spark incorrectly interpret partition name ending with 'd' or 'f' as number when reading partitioned parquets

user_825860 0
Viewed 232

I'm using spark.read.parquet() to read from a folder where parquet files are organized in partitions. The result will be wrong when the partition name ends with f or d. Apparently, Spark will interpret them as number instead of string. I have created a minimal test case as below to reproduce the problem.

df = spark.createDataFrame([
            ('9q', 1),
            ('3k', 2),
            ('6f', 3),
            ('7f', 4),
            ('7d', 5),
     ],
     schema='foo string, id integer'
)
df.write.partitionBy('foo').parquet('./tmp_parquet', mode='overwrite')
read_back_df = spark.read.parquet('./tmp_parquet')
read_back_df.show()

The read_back_df will be

+---+---+                                                                       
| id|foo|
+---+---+
|  1| 9q|
|  4|7.0|
|  3|6.0|
|  2| 3k|
|  5|7.0|
+---+---+

Notice partition 6f/7f/7d becomes 6.0/7.0/7.0.

The spark version is 2.4.3.

pyspark apache-spark-sql apache-spark
1 Answers

The behaviour that you see is expected.

From the Spark documentation:

Notice that the data types of the partitioning columns are automatically inferred.

You can disable this feature by setting spark.sql.sources.partitionColumnTypeInference.enabled to False.

The following code preserves the strings when reading the parquet file:

spark.conf.set("spark.sql.sources.partitionColumnTypeInference.enabled", False)
read_back_df = spark.read.parquet('./tmp_parquet')
read_back_df.show()

prints

+---+---+                                                                       
| id|foo|
+---+---+
|  3| 6f|
|  1| 9q|
|  4| 7f|
|  2| 3k|
|  5| 7d|
+---+---+
user_21298010
Related