How to use DB connection parameters in Glue job

user_16772185 0
Viewed 398

I'm trying to read files from S3 bucket and load them to RDS DB. I have used AWS Glue Studio and configured a job to run a python script for this load. I have also established a connection between Glue and RDS and added this connection in Job Details.

However, I had to define connection parameters again in the python script to establish connection to RDS like below.

conn = pg8000.connect(host,dbname,user,password);
cur = conn.cursor();

The job is running perfectly fine but I'm wondering is there a way to use the connection given in the Job details instead of hardcoding them in the script.

aws-glue amazon-rds amazon-web-services
1 Answers

If you already have a successful established Glue connection, you don't have to reconnect again in your python script with connect function.

You can simply use DynamicFrameReader class from the glue library to retrieve the data from your database as follows:

datasource = glue_context.create_dynamic_frame.from_catalog(
    database=<database_name>, table_name=<table_name>, transformation_ctx="datasource"

It returns a dynamicFrame which can be converted in dataframe as well.

You can read more about that : DynamicFrameReader

user_8704830
Related