How to pass dateformat value to spark job jar by CLI

Viewed 12

i have a spark job, builded using by sbt to jar when i spark-submit.

I want to pass param has space like yyyy-MM-dd HH:mm:ss, that is one param but CLI understand this is two. How i fix it?

spark-submit --class <className> --master local <jar path> <agr0:file path> yyyy-MM-dd dd-MM-yyyy string

here my code

 val logFile = args(0)
 val data = spark.sparkContext.textFile(logFile)
 ...
 val formatInputType = args(1)
 val requiredOutputFormat = args(2)
 val formatOutputType = args(3)
 val ds3 = ds2
      .withColumn("formatInputType", lit(formatInputType)) // "yyyy-MM-dd HH:mm:ss" ??
      .withColumn("requiredOutputFormat", lit(requiredOutputFormat)) // "dd-MM HH" ??
      .withColumn("formatOutputType", lit(formatOutputType)) // "epoch/string"
0 Answers
Related