AWS Glue Timeout: Creating External Schema In Redshift

user_16584537 0
Viewed 510

I am trying to create an external schema, and my command is as follows. As of course, I have changed the names of the components/items to non-meaningful names just to hide my production values:

create external schema sb_external 
from data catalog 
database 'dev' 
iam_role 'arn:aws:iam::490412345678:role/aws-service-role/redshift.amazonaws.com/AWSServiceRoleForRedshift'
create external database if not exists;

The query is ran in the Redshift database using "psql" CLI from within an EC2 instance. It is a private subnet, while the EC2 instance and the Redshift Database are in 2 different VPCs joined by VPC Peering. On the VPC where we have the EC2 instance, we have a Glue Endpoint.

While I run the above query from the same VPC where I have the Redshift database, I still get an error as follows, even if in the same VPC I have created an Endpoint Interface for Glue.

Failed to perform AWS request, curlError=Failed to connect to glue.eu-west-1.amazonaws.com port 443: Connection timed out

With or Without the VPC Endpoint, we have the same error.

Any help in this regard would be highly appreciated.

amazon-redshift aws-glue amazon-web-services
3 Answers

I have also faced the same issue and somehow I managed to resolve it. This error caused when you enable Enhanced VPC routing in your cluster.

By default Glue endpoint uses default security group. As error starting "glue.eu-west-1.amazonaws.com", you need to enable DNS hostnames and DNS resolution for your VPC. Also add inbound rule for port number 443 which is for https in default security group with source as Redshift's security group.

listing few links which helped me:

[+]. https://docs.aws.amazon.com/glue/latest/dg/vpc-interface-endpoints.html

[+]. https://docs.aws.amazon.com/vpc/latest/privatelink/create-interface-endpoint.html#vpce-interface-limitations

[+]. https://docs.aws.amazon.com/redshift/latest/mgmt/spectrum-enhanced-vpc.html#spectrum-enhanced-vpc-considerations

"Access to AWS Glue or Amazon Athena Redshift Spectrum accesses your data catalog in AWS Glue or Athena. Another option is to use a dedicated Hive metastore for your data catalog.

To enable access to AWS Glue or Athena, configure your VPC with an internet gateway or NAT gateway. Configure your VPC security groups to allow outbound traffic to the public endpoints for AWS Glue and Athena. Alternatively, you can configure an interface VPC endpoint for AWS Glue to access your AWS Glue Data Catalog. When you use a VPC interface endpoint, communication between your VPC and AWS Glue is conducted within the AWS network."

user_194447950
user_47250740

The security group associated with the Redshift cluster needs to have egress configured for enabling outbound traffic.

Example egress configuration:

  • from port: 0
  • to port: 0
  • protocol: -1 (all protocols)
  • CIDR IP: "0.0.0.0/0"

References

  1. AWS::EC2::SecurityGroupEgress
user_14360310
Related