I have a python function that connects to S3, and gets a file. This python script will run in the environments:
- DEV
- TEST
- PROD
For this I want to have a variable that stores the name of the bucket corresponding to the environment, which have the following names:
- bucket-dev00
- bucket-tst00
- bucket-prod00
How can I make this dynamic?
This python script is going to run in AWS Lambda or AWS Data Pipeline. I don't have it defined yet.
import pandas as pd
import boto3
from botocore.exceptions import ClientError
AWS_S3_BUCKET = "bucket-dev00"
def connect_s3(profile):
"""
connection to AWS S3
"""
session = boto3.Session(profile_name=profile)
s3_client = session.client('s3', region_name='us-east-1')
logger.info(f'S3 session successfully created')
return s3_client
def get_file_from_s3(profile):
"""
consume the corresponding file and error handling in case they occur
"""
s3_client = connect_s3(profile)
try:
response = s3_client.get_object(Bucket=AWS_S3_BUCKET, Key="folder1/file.csv")
except ClientError as e:
raise e
else:
status = response.get("ResponseMetadata", {}).get("HTTPStatusCode")
if status == 200:
df = pd.read_csv(response.get("Body"), sep=',', quotechar='"', skipinitialspace=True)
logger.info(f'Successful S3 get_object response. Status - {status}')
return df
else:
logger.info(f'Unsuccessful S3 get_object response. Status - {status}')