I am running spark master and worker in kubernetis container.I am sumbitting job using java(scala).Spark worker have log4j.properties.template file in conf folder as below. Here it specifies console Appender. Now all worker logs gets redirected to console but all drivers logs gets redirected to stderr file in work folder. In driver application jar I have used console appender.
# Set everything to be logged to the console
log4j.rootCategory=INFO, console
log4j.appender.console=org.apache.log4j.ConsoleAppender
log4j.appender.console.target=System.err
log4j.appender.console.layout=org.apache.log4j.PatternLayout
log4j.appender.console.layout.ConversionPattern=%d{yy/MM/dd HH:mm:ss} %p
%c{1}: %m%n
Could anyone please help me to redirect driver logs to console instead of sterr file