Check if running in Glue Job environment or Jupyter Notebook

user_6485881 0
Viewed 163

I need to do some setup in my Glue-ETL Job when the script is running inside of a Glue Job as opposed to my Jupyter Notebook.

Let's pretend my setup looks like this:

from awsglue.utils import getResolvedOptions
from pyspark.context import SparkContext
from awsglue.context import GlueContext

SC = SparkContext.getOrCreate()
GC = GlueContext(SC)

# I only want to run this when inside a Glue Job
args = getResolvedOptions(sys.argv, ['JOB_NAME'])
job = Job(GC)
job.init(args['JOB_NAME'], args)

I'm using a Glue (Jupyter) Notebook to develop my ETL script and export that to a Python script whenever I want to deploy it. So far I commented out the lines above for development and commented them back in for the deployment.

That's error-prone and not elegant.

Is there an easy way to discover whether I'm running inside of a Glue Job?

aws-glue-spark aws-glue pyspark amazon-web-services python
1 Answers

I have been using below solution for a while:

try:
    from awsglue.utils import getResolvedOptions
    is_glue_job = True
    #rest of the statements specific to glue
except:
    is_glue_job = False
    #rest of the statements

Here, try block is specific to glue only. awsglue library is available only in glue as of now. If code not running on glue, it will raise exception and reach except block.

user_91761140
Related