When I run spark-submit it works successfully, but the output is not colourised.
(/Users/me/bai/conda-envs/spark-mllib-kmeans) me@my-mbp spark-mllib-kmeans % spark-submit spark-helloWorld.py
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.apache.spark.unsafe.Platform (file:/usr/local/Cellar/apache-spark/3.0.1/libexec/jars/spark-unsafe_2.12-3.0.1.jar) to constructor java.nio.DirectByteBuffer(long,int)
WARNING: Please consider reporting this to the maintainers of org.apache.spark.unsafe.Platform
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
20/12/22 12:18:33 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
20/12/22 12:18:34 INFO SparkContext: Running Spark version 3.0.1
20/12/22 12:18:34 INFO ResourceUtils: ==============================================================
20/12/22 12:18:34 INFO ResourceUtils: Resources for spark.driver:
20/12/22 12:18:34 INFO ResourceUtils: ==============================================================
20/12/22 12:18:34 INFO SparkContext: Submitted application: Simple App
...
I am using Spark version 3.0.1:
(base) me@my-mbp spark-mllib-kmeans % spark-shell --version
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.apache.spark.unsafe.Platform (file:/usr/local/Cellar/apache-spark/3.0.1/libexec/jars/spark-unsafe_2.12-3.0.1.jar) to constructor java.nio.DirectByteBuffer(long,int)
WARNING: Please consider reporting this to the maintainers of org.apache.spark.unsafe.Platform
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
Welcome to
____ __
/ __/__ ___ _____/ /__
_\ \/ _ \/ _ `/ __/ '_/
/___/ .__/\_,_/_/ /_/\_\ version 3.0.1
/_/
Using Scala version 2.12.10, OpenJDK 64-Bit Server VM, 14.0.1
Branch HEAD
Compiled by user ubuntu on 2020-08-28T08:58:35Z
Revision 2b147c4cd50da32fe2b4167f97c8142102a0510d
Url https://gitbox.apache.org/repos/asf/spark.git
Type --help for more information.
I am using the default Mac terminal program on latest Mac OS:
% uname -a
Darwin my-mbp.lan 20.2.0 Darwin Kernel Version 20.2.0: Wed Dec 2 20:39:59 PST 2020; root:xnu-7195.60.75~1/RELEASE_X86_64 x86_64
I would like to see the different log statement levels (WARN/INFO/ERROR) in different colours. Perhaps other use of colours to differentiate the output from spark framework and the output from my application.
Given there is so much framework level output and there is noise of WARNINGS due to framework issues, I was hoping better use of colour could help me to scan my output quicker.
Is there a simple solution for this?
I see this behaviour in both native Mac Terminal and MS VSC integrated terminal.
I saw the output line:
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
I see I can create then edit the log4j config file here, so perhaps I just need the config for the log4j config file to colourise all output.
% cd $SPARK_HOME/conf
% cp log4j.properties.template log4j.properties