I am using GCP cloud composer ( Airflow ) to sync s3 files with GCS bucket. When I am setting up public composer (public cluster), I am able to run command “gsutil ls s3://bucket_name" and this list out files in it but when i setting up private cluster, I am getting connection timeout for the same. I am able to list google buckets content but unable to connect with s3 bucket…could someone help here
2 Answers
If GCP composer created a private GKE cluster, this cluster is isolated by design from the outside internet.
You should not be able to access resources outside Google Cloud such as S3 from inside the cluster, as is. There are options to enable outside connectivity, e.g., a NAT gateway.
Private Clusters don't have connectivity to resources outside of the cluster, what you need to do is to create a NAT Gateway and bind it to your cluster, that will allow you to reach resources in other clouds such as S3 in AWS, or allow your cluster to be whitelisted on other public facing services.
Related