my log4j.properties are packaged with the uber/fat application jar , here is my spark submit command
spark-submit --files file:log4j.properties
--conf spark.driver.extraJavaOptions=-Dlog4j.configuration=classpath:log4j.properties
--conf spark.executor.extraJavaOptions=-Dlog4j.configuration=classpath:log4j.properties
--class com.myproject.Application --deploy-mode cluster --master yarn s3://myappbucket/application.jar
the log4j.properties are packaged and available in the jar , but when running the application they are not available to the executors , I am facing the below error , what am I missing here , how do I get the log4j.properties available to the executors/worker nodes , would someone please help
I have used the classpath:log4j.properties in the spark submit command it looks like the executors are not taking my log4j configuration changes below
log4j.logger.com.myproject.Application=INFO,appfile
log4j.appender.appfile=org.apache.log4j.RollingFileAppender
log4j.appender.appfile.File=${base.log.dir}/app.log
log4j.appender.appfile.MaxFileSize=200MB
log4j.appender.appfile.MaxBackupIndex=10
Nothing is taking in to effect and everything is getting printed on the stdout and stderr log ,instead of creating the app.log and logging in the app.log file
