How to get the name of an s3 bucket depending on the environment where it is running? Python

Viewed 31

I have a python function that connects to S3, and gets a file. This python script will run in the environments:

  • DEV
  • TEST
  • PROD

For this I want to have a variable that stores the name of the bucket corresponding to the environment, which have the following names:

  • bucket-dev00
  • bucket-tst00
  • bucket-prod00

How can I make this dynamic?

This python script is going to run in AWS Lambda or AWS Data Pipeline. I don't have it defined yet.

import pandas as pd
import boto3
from botocore.exceptions import ClientError

AWS_S3_BUCKET = "bucket-dev00"

def connect_s3(profile):
    """
        connection to AWS S3
    """
    session = boto3.Session(profile_name=profile)

    s3_client = session.client('s3', region_name='us-east-1')
    logger.info(f'S3 session successfully created')

    return s3_client

def get_file_from_s3(profile):
    """
        consume the corresponding file and error handling in case they occur
    """
    s3_client = connect_s3(profile)
    try:
        response = s3_client.get_object(Bucket=AWS_S3_BUCKET, Key="folder1/file.csv")
    except ClientError as e:
        raise e
    else:
        status = response.get("ResponseMetadata", {}).get("HTTPStatusCode")

        if status == 200:
            df = pd.read_csv(response.get("Body"), sep=',', quotechar='"', skipinitialspace=True)
            logger.info(f'Successful S3 get_object response. Status - {status}')
            return df
        else:
            logger.info(f'Unsuccessful S3 get_object response. Status - {status}')

1 Answers

Use a dict (dictionary):

BUCKET_NAMES = dict(
    dev = "bucket-dev",
    test = "bucket-test",
    prod = "bucket-prod",
)

def foo():
    environment = "dev"
    bucket_name = BUCKET_NAMES[environment] # "bucket-dev"
Related