How to control Spark logging in IntelliJ

user_6145555 0
Viewed 846

I'm usually running stuff from JUnit but I've also tried running from main and it makes no difference.


I have read nearly two dozen SO questions, blog posts, and articles and tried almost everything to get Spark to stop logging so much.


Things I've tried:

  • log4j.properties in resources folder (in src and test)
  • Using spark-submit to add a log4j.properties which failed with "error: missing application resources"
  • Logger.getLogger("com").setLevel(Level.WARN);
  • Logger.getLogger("org").setLevel(Level.WARN);
  • Logger.getLogger("akka").setLevel(Level.WARN);
  • Logger.getRootLogger().setLevel(Level.WARN);
  • spark.sparkContext().setLogLevel("WARN");

    • In another project I got the logging to be quiet with:

    Logger.getLogger("org").setLevel(Level.WARN);
Logger.getLogger("akka").setLevel(Level.WARN);

    but it is not working here.


    How I'm creating my SparkSession:

    SparkSession spark = SparkSession
    .builder()
    .appName("RS-LDA")
    .master("local")
    .getOrCreate();


    Let me know if you'd like to see more of my code.

    Thanks

    apache-spark intellij-idea log4j java
    2 Answers

    I'm using IntelliJ and Spark and, this work for me:

    Logger.getRootLogger.setLevel(Level.ERROR)

    You could change Log Spark configurations too.

    $ cd SPARK_HOME/conf
$ gedit log4j.properties.template

# find this lines in the file
# Set everything to be logged to the console
log4j.rootCategory=INFO, console

and change to ERROR

log4j.rootCategory=ERROR, console

In this file you have other options tho change too

# Set the default spark-shell log level to WARN. When running the spark-shell, the
# log level for this class is used to overwrite the root logger's log level, so that
# the user can have different defaults for the shell and regular Spark apps.
log4j.logger.org.apache.spark.repl.Main=WARN

# Settings to quiet third party logs that are too verbose
.....

And finally rename the log4j.properties.template file
 $ mv log4j.properties.template log4j.properties

    You can follow this link for further configuration:

    Logging in Spark with Log4j

    or this one too:

    Logging in Spark with Log4j. How to customize the driver and executors for YARN cluster mode.

    user_85714980

    It might be an old question, but I just ran by the same problem. To fix it what I did was:

    1. Adding private static Logger log = LoggerFactory.getLogger(Spark.class);
      as a field for the class.
    2. spark.sparkContext().setLogLevel("WARN"); after creating the spark session

    Step 2 will work only after step 1.

    user_123608260
    Related