how to detect if your code is running under pyspark

user_149741 0
Viewed 590

For staging and production, my code will be running on PySpark. However, in my local development environment, I will not be running my code on PySpark.

This presents a problem from the standpoint of logging. Because one uses the Java library Log4J via Py4J when using PySpark, one will not be using Log4J for the local development.

Thankfully, the API for Log4J and the core Python logging module are the same: once you get a logger object, with either module you simply debug() or info() etc.

Thus, I wish to detect whether or not my code is being imported/run in PySpark or a non-PySpark environment: similar to:

class App:

    def our_logger(self):
        if self.running_under_spark():
            sc = SparkContext(conf=conf) 
            log4jLogger = sc._jvm.org.apache.log4j 
            log = log4jLogger.LogManager.getLogger(__name__) 
            log.warn("Hello World!")
            return log
        else:
            from loguru import logger
            return logger

How might I implement running_under_spark()

Simply trying to import pyspark and seeing if it works is not a fail-proof way of doing this because I have pyspark in my dev environment to kill warnings about non-imported modules in the code from my IDE.

pyspark
1 Answers

Maybe you can set some environment variable in your spark environment that you check for at runtime ( in $SPARK_HOME/conf/spark-env.sh):

export SPARKY=spark

Then you check if SPARKY exists to determine if you're in your spark environment.

from os import environ

class App:
def our_logger(self):
    if environ.get('SPARKY') is not None:
        sc = SparkContext(conf=conf) 
        log4jLogger = sc._jvm.org.apache.log4j 
        log = log4jLogger.LogManager.getLogger(__name__) 
        log.warn("Hello World!")
        return log
    else:
        from loguru import logger
        return logger
user_12661090
Related