I am using SparkLauncher in order to submit spark applications. I would like to redirect log information to my log file defined by logback.xml.I am using com.typesafe.scalalogging.LazyLogging.
This how I tried to do it and it's not working:
val sparkLauncher: SparkLauncher = new SparkLauncher(envVariable)
.setAppName("spark")
.redirectToLog(this.getClass.getName)
.setSparkHome(sparkInput.sparkHomePath)
.setMaster(sparkInput.master)
.setMainClass(sparkInput.classPath)
.setAppResource(sparkInput.jarPath)
.setVerbose(true)
val sparkLauncher: SparkLauncher = buildSparkLauncher(sparkInput)
val p = sparkLauncher.launch()
p.waitFor()