How to run an AWS Glue 1.0 Python Shell Job with the Current Version of boto3?

user_13644535 0
Viewed 1275

I need to use a newer boto3 package for an AWS Glue Python3 shell job (Glue Version: 1.0). I included the wheel file below from S3 as external Python Library:

boto3-1.13.21-py2.py3-none-any.whl

However, boto3.__version__ prints out 1.9.203 even if i see the following log:

Successfully installed boto3-1.13.21 botocore-1.16.26 docutils-0.15.2 jmespath-0.10.0 python-dateutil-2.8.1 s3transfer-0.3.3 six-1.15.0 urllib3-1.25.10

For some reason, Glue Python Shell job is not letting me overwrite the boto3 package version with the wheel file. Is there any way to overwrite?

aws-glue boto3 python
2 Answers

The reason for this happening is that boto3 is already imported when the GLue job starts, therefore, the import statement is ignored by Python. You need to remove the imported package at the beginning of your Glue script. You need to insert following code to the top of the script:

import sys
sys.path.insert(0, '/glue/lib/installation')
keys = [k for k in sys.modules.keys() if ('boto3' in k)]
for k in keys:
    del sys.modules[k]


import boto3
print(boto3.__version__)
user_170722370

Steps to run an AWS Glue 1.0 Python Shell Job with the Current Version of boto3.

  1. Download the current versions of .whl files from pip.
  1. Create a bucket and folder where you want to store python whl files that you might use for glue python jobs.
  • s3://glue/python/lib/
  1. Upload the boto3 and botocore whl files to the new folder in the s3 bucket created in step2.
  • s3://glue/python/lib/botocore-1.23.10-py3-none-any.whl,
  • s3://glue/python/lib/boto3-1.20.10-py3-none-any.whl

  1. Create a Glue 1.0 python shell job and while doing so expand Security configuration, in the "Python library path" add the paths with a comma separating the the paths

    s3://glue/python/lib/botocore-1.23.10-py3-none-any.whl,s3://glue/python/lib/boto3-1.20.10-py3-none-any.whl

  2. Create a new python script and at the top of the script have it remove the pre-imported boto3 package.

    import sys
sys.path.insert(0, '/glue/lib/installation')
keys = [k for k in sys.modules.keys() if 'boto' in k]
for k in keys:
    if 'boto' in k:
        del sys.modules[k]

  3. Import boto3

    import boto3

This is what my final template usually looks like

import sys
sys.path.insert(0, '/glue/lib/installation')
keys = [k for k in sys.modules.keys() if 'boto' in k]
for k in keys:
    if 'boto' in k:
        del sys.modules[k]

import logging
import boto3
import botocore


def main():
    handler = logging.StreamHandler(sys.stdout)
    formatter = logging.Formatter('%(asctime)s-%(name)s-%(levelname)s-%(process)d: %(message)s')
    handler.setFormatter(formatter)
    logger = logging.getLogger("GlueLogger")
    logger.setLevel(logging.INFO)
    logger.addHandler(handler)
    logger.info(f"Received sys.argv: {sys.argv}")
    logger.info(f"Using boto3 api version:{boto3.__version__}")


if __name__ == "__main__":
    main()
user_12848680
Related