How to write partitioned DataFrame out without partition prefix in the path?

user_5686654 0
Viewed 883

I have code:

dataFrame.write
  .partitionBy("format", "dataset", "year", "month", "day", "hour")
  .format(outputFormat)
  .mode("overwrite")
  .save(outputPath)

that gives an output like: enter image description here

How can I write dataFrame without prefixes ? Not renaming. Is there any way to do this by Spark or hadoop configurations? Currently I'm using separate bash-script to rename folders.

To be clear, I want to get next structure using Spark code:

data
-+parquet
   +main
     +2017
        +01
         +31
           +15
apache-spark-sql apache-spark scala
1 Answers

You can't do this.

If you look at Spark code, class ExternalCatalogUtils.getPartitionPathString, you see:

def getPartitionPathString(col: String, value: String): String = {
    val partitionString = if (value == null || value.isEmpty) {
      DEFAULT_PARTITION_NAME
    } else {
      escapePathName(value)
    }
    escapePathName(col) + "=" + partitionString
}

So always the scheme will be like column=value

user_58118000
Related