apache-spark partitionBy: remove column names from directory layout

user_1344509 0
Viewed 779

I have code like this:

val data1 = data.withColumn("local_date_time", toLocalDateUdf('timestamp))
data1
  .withColumn("year", year(col("local_date_time")))
  .withColumn("month", month(col("local_date_time")))
  .withColumn("day", dayofmonth(col("local_date_time")))
  .withColumn("hour", hour(col("local_date_time")))
  .drop("local_date_time")
  .write
  .mode("append")
  .partitionBy("year", "month", "day", "hour")
  .format("json")
  .save("s3a://path/")

It creates nested folders like this year=2020 / month=5 / day=10 is S3 (year is column name, 2020 is its value). I want to create nested folders like 2020 / 5 / 10. Spark is adding column name to directory name if I use partitionBy method.

This is from Spark source code:

  /**
   * Partitions the output by the given columns on the file system. If specified, the output is
   * laid out on the file system similar to Hive's partitioning scheme. As an example, when we
   * partition a dataset by year and then month, the directory layout would look like:
   * <ul>
   * <li>year=2016/month=01/</li>
   * <li>year=2016/month=02/</li>
   * </ul>
   */
    @scala.annotation.varargs
    def partitionBy(colNames: String*): DataFrameWriter[T] = {
      this.partitioningColumns = Option(colNames)
      this
    }

How can I remove column names from directory layout?

apache-spark scala
1 Answers

.partitionBy("year", "month", "day", "hour")

Command above allows you to save it into parquet with partitions in partition=value format

This is not a bug, it's a standard parquet format.

You can loop over each partition and save it manually otherwise

user_89045520
Related