In Spark I did wrote a table like this (I did this because first table had too many partitions):
spark.table('table1')\
.coalesce(2000).write.format('orc').mode('overwrite').saveAsTable("table2")
When i'm looking in hdfs, i'm seeing this:
hdfs dfs -ls hdfs://ha/apps/hive/warehouse/my.db/table2
Found 2001 items
That's correct.
But when I'm trying to use getNumPartitions, then I see something strange:
spark.table('table2').rdd.getNumPartitions()
1962
I was tried to do repartition(2000) instead of coalesce, but:
spark.table('table1')\
.repartition(2000).write.format('orc').mode('overwrite').saveAsTable("table3")
hdfs dfs -ls hdfs://ha/apps/hive/warehouse/my.db/table3
Found 2001 items
spark.table('table3').rdd.getNumPartitions()
2167
Just for interesting, i make repartition on specific column, which contains only 293 unique value, and that's the result:
spark.table('table1')\
.repartition(2000, "agg_date").write.format('orc').mode('overwrite').saveAsTable("table4")
hdfs dfs -ls hdfs://ha/apps/hive/warehouse/my.db/table4
Found 276 items
spark.table('table4').rdd.getNumPartitions()
2167
I cant't explain, why it's working so strange. Why in the first and second cases I've got correct number of files, bus strange results of getNumPartitions()? And why in the third case I was get 275 files instead of 293 that I was expected and getNumPartitions() show me something weird?
Thank you!