PySpark 2.4.4 was installed using pip on a Windows machine.
SparkSession is used to bootstrap a pyspark session with master set to local.
Everything works as anticipated with the exception of overriding the default log4j properties.
I've tried setting the spark conf directory. Adding the conf directory to the spark home directory with properties file. Setting extra config options in the SparkSession but nothing seems to make any progress.
No matter what I do, log4j.proprerties appears to be ignored.
How can I setup my spark session to use a override log4j.properties file...