ETL Connector not loading in aws

user_7773898 0
Viewed 2022

I am using bigquery connector for aws glue in my glue job. It was working fine few days ago but now suddenly it is giving me below error:

LAUNCH ERROR | Glue ETL Marketplace - failed to download connector.Please refer logs for details.

Below is the full error that i am getting on cloudwatch


2021-11-08T11:33:02.045+05:00   Traceback (most recent call last): File "/usr/lib64/python3.7/runpy.py", line 193, in _run_module_as_main

2021-11-08T11:33:02.070+05:00   "__main__", mod_spec) File "/usr/lib64/python3.7/runpy.py", line 85, in _run_code exec(code, run_globals) File "/tmp/aws_glue_custom_connector_python/docker/unpack_docker_image.py", line 361, in <module>

2021-11-08T11:33:02.070+05:00   main() File "/tmp/aws_glue_custom_connector_python/docker/unpack_docker_image.py", line 351, in main

2021-11-08T11:33:02.070+05:00   res += download_jars_per_connection(conn, region, endpoint, proxy) File "/tmp/aws_glue_custom_connector_python/docker/unpack_docker_image.py", line 304, in download_jars_per_connection

2021-11-08T11:33:02.070+05:00   download_and_unpack_docker_layer(ecr_url, layer["digest"], dir_prefix, http_header) File "/tmp/aws_glue_custom_connector_python/docker/unpack_docker_image.py", line 168, in download_and_unpack_docker_layer

2021-11-08T11:33:02.070+05:00   layer = send_get_request(layer_url, header) File "/tmp/aws_glue_custom_connector_python/docker/unpack_docker_image.py", line 80, in send_get_request

2021-11-08T11:33:02.070+05:00   

2021-11-08T11:33:02.070+05:00   response.raise_for_status() File "/home/spark/.local/lib/python3.7/site-packages/requests/models.py", line 765, in raise_for_status

2021-11-08T11:33:02.071+05:00   raise HTTPError(http_error_msg, response=self) requests.exceptions.HTTPError: 400 Client Error: Bad Request

2021-11-08T11:33:02.119+05:00   Glue ETL Marketplace - failed to download connector, activation script exited with code
apache-spark amazon-web-services
4 Answers

When Glue job tried using connector, it has to download the connector in the form of container. The containers for connectors are available in amazon public ECR repo. To pull the container from the AWS public repo, we have to add "AmazonEC2ContainerRegistryFullAccess" policy to your IAM role. We can limit the access to read only as well.

user_102202530

This could very likely be a permissions issue. I was running into it and temporarily gave looser permissions, which seemed to resolve it.

user_140280130

I've come around this in an organizational setting trying to use the BigQuery Markertplace connector. I was explicitly denied to GetAuthorizationToken on non eu regions. Hence, the Glue Job will fail in a similar way as describes by OP because it tries to download the docker image on run from here: https://709825985650.dkr.ecr.us-east-1.amazonaws.com/amazon-web-services/glue/bigquery:0.22.0-glue3.0-2

A possible workaround is to push a copy of the image to your private ECR. Then, when creating the GLUE connection, set CONNECTOR_URL in its connection_properties to your private ECR url. This will solve similar issues. Also, this seems more reasonable then adding wide reaching policies like AmazonEC2ContainerRegistryFullAccess (as suggested by Sparkian) to a role. You'll be able to give granular access permissions on this specific ECR repo instead.

user_52510610

I have this issue with the GCP BigQuery connector. Some jobs are able to run the connector, some aren't. All have the same permissions and settings. There seems to be an issue after requesting the ECR Authorization Token where the request times out.

user_182713330
Related