Is it possible to use one AWS Glue job to write data into different databases?

user_441874 0
Viewed 1150

I am planning to use AWS Glue to do a simple job of extracting data from AWS s3 bucket and load it into RDS database. The plan is to use AWS wizard to create a python script, with minimum modifications.

The problem is that I need to load pretty much the same data into many separated DBs, which has the same schema, same tables, etc.

The question is if it possible to do it with one glue script (using several connections in "Required connections" in wizard, for example), or do I need to have a separate glue job for each DB?

aws-glue
1 Answers

It looks like it's possible only under certain conditions (I found it here https://forums.aws.amazon.com/thread.jspa?threadID=287775): "When you use a JDBC connection as the data source, an ENI is launched in the subnet which is defined in the Connection. Glue resources uses this ENI for all of the traffic to your data sources. When you add multiple connections to a job, it will always launch the ENI in the subnet specified with the first connection that is added to the job." When I try it with just 2 different connections with different subnets, so only one works, and another fails with "Connection attempt timed out" error - as described in provided link.

user_4418740
Related