Spark: PartitionBy, change output file name

Viewed 3123

Currently, when I use the paritionBy() to write to HDFS:

DF.write.partitionBy("id")

I will get output structure looking like (which is the default behaviour):

../id=1/

../id=2/

../id=3/

I would like a structure looking like:

../a/

../b/

../c/

such that:

if id = 1, then a
if id = 2, then b

.. etc

Is there a way to change the filename output? If not, what is the best way to do this?

1 Answers
Related