Number of files in hdfs not equal result of getNumPartitions, how it is possible?

Viewed 129

In Spark I did wrote a table like this (I did this because first table had too many partitions):

spark.table('table1')\
.coalesce(2000).write.format('orc').mode('overwrite').saveAsTable("table2")

When i'm looking in hdfs, i'm seeing this:

hdfs dfs -ls hdfs://ha/apps/hive/warehouse/my.db/table2
Found 2001 items

That's correct. But when I'm trying to use getNumPartitions, then I see something strange:

spark.table('table2').rdd.getNumPartitions()
1962

I was tried to do repartition(2000) instead of coalesce, but:

spark.table('table1')\
.repartition(2000).write.format('orc').mode('overwrite').saveAsTable("table3")

hdfs dfs -ls hdfs://ha/apps/hive/warehouse/my.db/table3
Found 2001 items

spark.table('table3').rdd.getNumPartitions()
2167

Just for interesting, i make repartition on specific column, which contains only 293 unique value, and that's the result:

spark.table('table1')\
    .repartition(2000, "agg_date").write.format('orc').mode('overwrite').saveAsTable("table4")

hdfs dfs -ls hdfs://ha/apps/hive/warehouse/my.db/table4
Found 276 items

spark.table('table4').rdd.getNumPartitions()
2167

I cant't explain, why it's working so strange. Why in the first and second cases I've got correct number of files, bus strange results of getNumPartitions()? And why in the third case I was get 275 files instead of 293 that I was expected and getNumPartitions() show me something weird?

Thank you!

0 Answers
Related