I have a spark dataframe with 2 columns - col1 and col2.
scala> val df = List((1, "a")).toDF("col1", "col2")
df: org.apache.spark.sql.DataFrame = [col1: int, col2: string]
When I write df on disk in parquet format, to write all the data in number of files equal to the number of unique values in col1 I do a repartition using col1, like this:
scala> df.repartition(col("col1")).write.partitionBy("col1").parquet("file")
Above code produces only one file in filesystem. But, the number of shuffle operations becomes 200.
I am not able to understand one thing here that if col1 contains only one value, i.e., 1 then why it is creating 200 partitions in repartition ?