AWS region in AWS Glue

user_303598 0
Viewed 3174

How can I get the region in which the current Glue job is executing?

When the Glue job starts executing, I see the output

Detected region eu-central-1.

In AWS Lambda, I can use the following lines to fetch the current region:

import os
region = os.environ['AWS_REGION']

However, it seems like the AWS_REGION environment variable is not present in Glue and therefore a KeyError is raised:

KeyError: 'AWS_REGION'

The reason why I need the region is I am trying to fetch all databases and tables as described in this question and I do not want to hard code the region when creating the boto client.

aws-glue amazon-web-services python
3 Answers

One option is to pass the AWS_REGION as a job parameter. For example, if you trigger the job from Lambda:

import os

response = client.start_job_run(
    JobName = 'a_job_name',
    Arguments = {'--AWS_REGION': os.environ['AWS_REGION'] } 
)

Alternatively, if you define your jobs using the AWS::Glue::Job CloudFormation resource:

GlueJob:
  Type: AWS::Glue::Job
  Properties:
    Role: !Ref GlueRole
    DefaultArguments:
      "--AWS_REGION": !Sub "${AWS::Region}"
    Command:
      ScriptLocation: !Sub s3://${GlueScriptBucket}/glue-job.py
      Name: glueetl

Then you can extract the AWS_REGION parameter in your job code using getResolvedOptions:

import sys
from awsglue.utils import getResolvedOptions

args = getResolvedOptions(sys.argv, ['AWS_REGION'])
print('region', args['AWS_REGION'])
user_3035980

Use os.environ['AWS_DEFAULT_REGION'] instead.

Leaving this here for new visitors.

user_27698030

os.environ['AWS_DEFAULT_REGION'] works for Glue versions 2.0 and 3.0, but does not exist in previous versions. It gives the region code, for example us-east-1.

This was confirmed through running a small PySpark script to print out the environment variables, on all the Glue versions, as suggested by the other answer.

user_65801420
Related