Does Google Cloud Composer support connecting to AWS resource like S3 and Redshift?

user_5972935 0
Viewed 1745

I am planning to use Google Cloud Composer to schedule a workflow which is loading data from S3 to Redshift. As S3 and Redshift are all based on AWS, I want to know whether Google Cloud Composer allows me to do it.

From my understanding after reading the document, I am able to install AWS SDK from PYPI and install local Python dependencies, which means that I can build my own library for S3Connection and RedshiftConnection to make the workflow work. Anyone with experience on it can help confirm it? Thanks

google-cloud-composer
1 Answers

Cloud Composer is built to support hybrid and multi-cloud as mentioned here.

Cloud Composer can make use of existing Airflow operators which includes a S3toRedshift operator that uses the S3 hook and the postgres hook for Redshift

Cloud Composer can also use custom plugins. The way to install custom plugins in Cloud Composer is described in this link.

user_4350890
Related