I'm using the AWS CLI on my server (my server isn't an EC2 instance) to list the contents of an open data set on an AWS S3, and it works fine:
$ aws --no-sign-request s3 ls s3://commoncrawl/cc-index/table/cc-main/warc/crawl=CC-MAIN-2013-20/
PRE subset=warc/
But when I use --debug to see what's happening behind the scenes, I see exceptions caused by timeouts, such as:
2020-06-02 14:08:14,945 - MainThread - botocore.utils - DEBUG - Caught retryable HTTP exception while making metadata service request to http://169.254.169.254/latest/api/token: Connect timeout on endpoint URL: "http://169.254.169.254/latest/api/token"
Traceback (most recent call last):
File "urllib3/connection.py", line 160, in _new_conn
File "urllib3/util/connection.py", line 84, in create_connection
File "urllib3/util/connection.py", line 74, in create_connection
socket.timeout: timed out
and:
2020-06-02 14:08:15,948 - MainThread - botocore.utils - DEBUG - Caught retryable HTTP exception while making metadata service request to http://169.254.169.254/latest/meta-data/placement/availability-zone/: Connect timeout on endpoint URL: "http://169.254.169.254/latest/meta-data/placement/availability-zone/"
Traceback (most recent call last):
File "urllib3/connection.py", line 160, in _new_conn
File "urllib3/util/connection.py", line 84, in create_connection
File "urllib3/util/connection.py", line 74, in create_connection
socket.timeout: timed out
It's working, in the sense that it's returning the proper response from the S3 bucket, but why do I see those exceptions in the bucket mode? Why does boto complain for the
- http://169.254.169.254/latest/api/token: Connect timeout on endpoint
- http://169.254.169.254/latest/meta-data/placement/availability-zone/: Connect timeout on endpoint
The version of AWS CLI that I use:
$ aws --version
aws-cli/2.0.17 Python/3.7.3 Linux/5.3.0-45-generic botocore/2.0.0dev21
I run AWS CLI on an Ubuntu GNU/Linux server 19.10.
Is there a way to prevent those exceptions? I'm asking this because I'm using the same aws cli utility to connect to non-AWS object stores, too. In this case there's no concept of "IAM" with respect to such S3-compatible object storage. This is why I'm trying to understand if I can make aws cli skip that part.
The full debug output:
$ aws --debug --no-sign-request s3 ls s3://commoncrawl/cc-index/table/cc-main/warc/crawl=CC-MAIN-2013-20/
2020-06-02 14:24:31,488 - MainThread - awscli.clidriver - DEBUG - CLI version: aws-cli/2.0.17 Python/3.7.3 Linux/5.3.0-45-generic botocore/2.0.0dev21
2020-06-02 14:24:31,489 - MainThread - awscli.clidriver - DEBUG - Arguments entered to CLI: ['--debug', '--no-sign-request', 's3', 'ls', 's3://commoncrawl/cc-index/table/cc-main/warc/crawl=CC-MAIN-2013-20/']
2020-06-02 14:24:31,489 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function add_timestamp_parser at 0x7ff2e8f5a598>
2020-06-02 14:24:31,489 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function register_uri_param_handler at 0x7ff2e98a6268>
2020-06-02 14:24:31,489 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function add_binary_formatter at 0x7ff2e8f1da60>
2020-06-02 14:24:31,489 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function inject_assume_role_provider_cache at 0x7ff2e985f840>
2020-06-02 14:24:31,491 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function attach_history_handler at 0x7ff2e90a40d0>
2020-06-02 14:24:31,491 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function inject_json_file_cache at 0x7ff2e90d89d8>
2020-06-02 14:24:31,491 - MainThread - botocore.hooks - DEBUG - Event building-command-table.s3: calling handler <function add_waiters at 0x7ff2e8f619d8>
2020-06-02 14:24:31,492 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.s3.anonymous: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7ff2e8e9f400>
2020-06-02 14:24:31,492 - MainThread - botocore.hooks - DEBUG - Event building-command-table.ls: calling handler <function add_waiters at 0x7ff2e8f619d8>
2020-06-02 14:24:31,493 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.ls.paths: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7ff2e8e9f400>
2020-06-02 14:24:31,493 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.ls.anonymous: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7ff2e8e9f400>
2020-06-02 14:24:31,493 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.ls.page-size: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7ff2e8e9f400>
2020-06-02 14:24:31,494 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.ls.human-readable: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7ff2e8e9f400>
2020-06-02 14:24:31,494 - MainThread - botocore.hooks - DEBUG - Event process-cli-arg.custom.ls: calling handler <awscli.argprocess.ParamShorthandParser object at 0x7ff2e9830940>
2020-06-02 14:24:31,494 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.ls.summarize: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7ff2e8e9f400>
2020-06-02 14:24:31,494 - MainThread - botocore.hooks - DEBUG - Event process-cli-arg.custom.ls: calling handler <awscli.argprocess.ParamShorthandParser object at 0x7ff2e9830940>
2020-06-02 14:24:31,494 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.ls.request-payer: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7ff2e8e9f400>
2020-06-02 14:24:31,496 - MainThread - urllib3.connectionpool - DEBUG - Starting new HTTP connection (1): 169.254.169.254:80
2020-06-02 14:24:32,498 - MainThread - botocore.utils - DEBUG - Caught retryable HTTP exception while making metadata service request to http://169.254.169.254/latest/api/token: Connect timeout on endpoint URL: "http://169.254.169.254/latest/api/token"
Traceback (most recent call last):
File "urllib3/connection.py", line 160, in _new_conn
File "urllib3/util/connection.py", line 84, in create_connection
File "urllib3/util/connection.py", line 74, in create_connection
socket.timeout: timed out
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "botocore/httpsession.py", line 263, in send
File "urllib3/connectionpool.py", line 725, in urlopen
File "urllib3/util/retry.py", line 379, in increment
File "urllib3/packages/six.py", line 735, in reraise
File "urllib3/connectionpool.py", line 677, in urlopen
File "urllib3/connectionpool.py", line 392, in _make_request
File "http/client.py", line 1229, in request
File "botocore/awsrequest.py", line 92, in _send_request
File "http/client.py", line 1275, in _send_request
File "http/client.py", line 1224, in endheaders
File "botocore/awsrequest.py", line 119, in _send_output
File "botocore/awsrequest.py", line 203, in send
File "http/client.py", line 956, in send
File "urllib3/connection.py", line 187, in connect
File "urllib3/connection.py", line 167, in _new_conn
urllib3.exceptions.ConnectTimeoutError: (<botocore.awsrequest.AWSHTTPConnection object at 0x7ff2e8e3b470>, 'Connection to 169.254.169.254 timed out. (connect timeout=1)')
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "botocore/utils.py", line 221, in _fetch_metadata_token
File "botocore/httpsession.py", line 287, in send
botocore.exceptions.ConnectTimeoutError: Connect timeout on endpoint URL: "http://169.254.169.254/latest/api/token"
2020-06-02 14:24:32,499 - MainThread - urllib3.connectionpool - DEBUG - Starting new HTTP connection (2): 169.254.169.254:80
2020-06-02 14:24:33,500 - MainThread - botocore.utils - DEBUG - Caught retryable HTTP exception while making metadata service request to http://169.254.169.254/latest/meta-data/placement/availability-zone/: Connect timeout on endpoint URL: "http://169.254.169.254/latest/meta-data/placement/availability-zone/"
Traceback (most recent call last):
File "urllib3/connection.py", line 160, in _new_conn
File "urllib3/util/connection.py", line 84, in create_connection
File "urllib3/util/connection.py", line 74, in create_connection
socket.timeout: timed out
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "botocore/httpsession.py", line 263, in send
File "urllib3/connectionpool.py", line 725, in urlopen
File "urllib3/util/retry.py", line 379, in increment
File "urllib3/packages/six.py", line 735, in reraise
File "urllib3/connectionpool.py", line 677, in urlopen
File "urllib3/connectionpool.py", line 392, in _make_request
File "http/client.py", line 1229, in request
File "botocore/awsrequest.py", line 92, in _send_request
File "http/client.py", line 1275, in _send_request
File "http/client.py", line 1224, in endheaders
File "botocore/awsrequest.py", line 119, in _send_output
File "botocore/awsrequest.py", line 203, in send
File "http/client.py", line 956, in send
File "urllib3/connection.py", line 187, in connect
File "urllib3/connection.py", line 167, in _new_conn
urllib3.exceptions.ConnectTimeoutError: (<botocore.awsrequest.AWSHTTPConnection object at 0x7ff2e8e3b748>, 'Connection to 169.254.169.254 timed out. (connect timeout=1)')
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "botocore/utils.py", line 264, in _get_request
File "botocore/httpsession.py", line 287, in send
botocore.exceptions.ConnectTimeoutError: Connect timeout on endpoint URL: "http://169.254.169.254/latest/meta-data/placement/availability-zone/"
2020-06-02 14:24:33,501 - MainThread - awscli.utils - DEBUG - Max number of attempts exceeded (1) when attempting to retrieve data from metadata service.
2020-06-02 14:24:33,501 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: env
2020-06-02 14:24:33,502 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: assume-role
2020-06-02 14:24:33,502 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: assume-role-with-web-identity
2020-06-02 14:24:33,502 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: sso
2020-06-02 14:24:33,502 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: shared-credentials-file
2020-06-02 14:24:33,503 - MainThread - botocore.credentials - INFO - Found credentials in shared credentials file: ~/.aws/credentials
2020-06-02 14:24:33,503 - MainThread - botocore.loaders - DEBUG - Loading JSON file: /usr/local/aws-cli/v2/2.0.17/dist/botocore/data/endpoints.json
2020-06-02 14:24:33,510 - MainThread - botocore.hooks - DEBUG - Event choose-service-name: calling handler <function handle_service_name_alias at 0x7ff2ea856bf8>
2020-06-02 14:24:33,517 - MainThread - botocore.loaders - DEBUG - Loading JSON file: /usr/local/aws-cli/v2/2.0.17/dist/botocore/data/s3/2006-03-01/service-2.json
2020-06-02 14:24:33,527 - MainThread - botocore.hooks - DEBUG - Event creating-client-class.s3: calling handler <function add_generate_presigned_post at 0x7ff2ea8a4bf8>
2020-06-02 14:24:33,527 - MainThread - botocore.hooks - DEBUG - Event creating-client-class.s3: calling handler <function add_generate_presigned_url at 0x7ff2ea8a49d8>
2020-06-02 14:24:33,532 - MainThread - botocore.endpoint - DEBUG - Setting s3 timeout as (60, 60)
2020-06-02 14:24:33,539 - MainThread - botocore.loaders - DEBUG - Loading JSON file: /usr/local/aws-cli/v2/2.0.17/dist/botocore/data/s3/2006-03-01/paginators-1.json
2020-06-02 14:24:33,540 - MainThread - botocore.hooks - DEBUG - Event provide-client-params.s3.ListObjectsV2: calling handler <function base64_decode_input_blobs at 0x7ff2e8f222f0>
2020-06-02 14:24:33,541 - MainThread - botocore.hooks - DEBUG - Event before-parameter-build.s3.ListObjectsV2: calling handler <function set_list_objects_encoding_type_url at 0x7ff2ea878158>
2020-06-02 14:24:33,541 - MainThread - botocore.hooks - DEBUG - Event before-parameter-build.s3.ListObjectsV2: calling handler <function validate_bucket_name at 0x7ff2ea874ea0>
2020-06-02 14:24:33,541 - MainThread - botocore.hooks - DEBUG - Event before-parameter-build.s3.ListObjectsV2: calling handler <bound method S3RegionRedirector.redirect_from_cache of <botocore.utils.S3RegionRedirector object at 0x7ff2e89b66d8>>
2020-06-02 14:24:33,541 - MainThread - botocore.hooks - DEBUG - Event before-parameter-build.s3.ListObjectsV2: calling handler <bound method S3ArnParamHandler.handle_arn of <botocore.utils.S3ArnParamHandler object at 0x7ff2e89b6a20>>
2020-06-02 14:24:33,541 - MainThread - botocore.hooks - DEBUG - Event before-parameter-build.s3.ListObjectsV2: calling handler <function generate_idempotent_uuid at 0x7ff2ea874ae8>
2020-06-02 14:24:33,542 - MainThread - botocore.hooks - DEBUG - Event before-call.s3.ListObjectsV2: calling handler <function add_expect_header at 0x7ff2ea877268>
2020-06-02 14:24:33,542 - MainThread - botocore.hooks - DEBUG - Event before-call.s3.ListObjectsV2: calling handler <bound method S3RegionRedirector.set_request_url of <botocore.utils.S3RegionRedirector object at 0x7ff2e89b66d8>>
2020-06-02 14:24:33,542 - MainThread - botocore.hooks - DEBUG - Event before-call.s3.ListObjectsV2: calling handler <function inject_api_version_header_if_needed at 0x7ff2ea878510>
2020-06-02 14:24:33,542 - MainThread - botocore.endpoint - DEBUG - Making request for OperationModel(name=ListObjectsV2) with params: {'url_path': '/commoncrawl?list-type=2', 'query_string': {'prefix': 'cc-index/table/cc-main/warc/crawl=CC-MAIN-2013-20/', 'delimiter': '/', 'encoding-type': 'url'}, 'method': 'GET', 'headers': {'User-Agent': 'aws-cli/2.0.17 Python/3.7.3 Linux/5.3.0-45-generic botocore/2.0.0dev21'}, 'body': b'', 'url': 'https://s3.amazonaws.com/commoncrawl?list-type=2&prefix=cc-index%2Ftable%2Fcc-main%2Fwarc%2Fcrawl%3DCC-MAIN-2013-20%2F&delimiter=%2F&encoding-type=url', 'context': {'client_region': 'aws-global', 'client_config': <botocore.config.Config object at 0x7ff2e89b62b0>, 'has_streaming_input': False, 'auth_type': None, 'encoding_type_auto_set': True, 'signing': {'bucket': 'commoncrawl'}}}
2020-06-02 14:24:33,542 - MainThread - botocore.hooks - DEBUG - Event request-created.s3.ListObjectsV2: calling handler <bound method RequestSigner.handler of <botocore.signers.RequestSigner object at 0x7ff2e89b6240>>
2020-06-02 14:24:33,542 - MainThread - botocore.hooks - DEBUG - Event choose-signer.s3.ListObjectsV2: calling handler <function set_operation_specific_signer at 0x7ff2ea8749d8>
2020-06-02 14:24:33,542 - MainThread - botocore.hooks - DEBUG - Event choose-signer.s3.ListObjectsV2: calling handler <function disable_signing at 0x7ff2ea8771e0>
2020-06-02 14:24:33,542 - MainThread - botocore.hooks - DEBUG - Event before-sign.s3.ListObjectsV2: calling handler <bound method S3EndpointSetter.set_endpoint of <botocore.utils.S3EndpointSetter object at 0x7ff2e89b6a90>>
2020-06-02 14:24:33,543 - MainThread - botocore.utils - DEBUG - Defaulting to S3 virtual host style addressing with path style addressing fallback.
2020-06-02 14:24:33,543 - MainThread - botocore.utils - DEBUG - Checking for DNS compatible bucket for: https://s3.amazonaws.com/commoncrawl?list-type=2&prefix=cc-index%2Ftable%2Fcc-main%2Fwarc%2Fcrawl%3DCC-MAIN-2013-20%2F&delimiter=%2F&encoding-type=url
2020-06-02 14:24:33,543 - MainThread - botocore.utils - DEBUG - URI updated to: https://commoncrawl.s3.amazonaws.com/?list-type=2&prefix=cc-index%2Ftable%2Fcc-main%2Fwarc%2Fcrawl%3DCC-MAIN-2013-20%2F&delimiter=%2F&encoding-type=url
2020-06-02 14:24:33,543 - MainThread - botocore.endpoint - DEBUG - Sending http request: <AWSPreparedRequest stream_output=False, method=GET, url=https://commoncrawl.s3.amazonaws.com/?list-type=2&prefix=cc-index%2Ftable%2Fcc-main%2Fwarc%2Fcrawl%3DCC-MAIN-2013-20%2F&delimiter=%2F&encoding-type=url, headers={'User-Agent': b'aws-cli/2.0.17 Python/3.7.3 Linux/5.3.0-45-generic botocore/2.0.0dev21'}>
2020-06-02 14:24:33,544 - MainThread - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): commoncrawl.s3.amazonaws.com:443
2020-06-02 14:24:33,988 - MainThread - urllib3.connectionpool - DEBUG - https://commoncrawl.s3.amazonaws.com:443 "GET /?list-type=2&prefix=cc-index%2Ftable%2Fcc-main%2Fwarc%2Fcrawl%3DCC-MAIN-2013-20%2F&delimiter=%2F&encoding-type=url HTTP/1.1" 200 None
2020-06-02 14:24:33,989 - MainThread - botocore.parsers - DEBUG - Response headers: {'x-amz-id-2': 'd83WTlBDKacMBr+0ReHHYaZmErezzMJxNvTChfrzCeF8gc9SFPcUGX59R7Qk3KSG3kpaK+OM/jI=', 'x-amz-request-id': 'AF67B94BB8A77A9F', 'Date': 'Tue, 02 Jun 2020 14:24:34 GMT', 'x-amz-bucket-region': 'us-east-1', 'Content-Type': 'application/xml', 'Transfer-Encoding': 'chunked', 'Server': 'AmazonS3'}
2020-06-02 14:24:33,989 - MainThread - botocore.parsers - DEBUG - Response body:
b'<?xml version="1.0" encoding="UTF-8"?>\n<ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Name>commoncrawl</Name><Prefix>cc-index/table/cc-main/warc/crawl%3DCC-MAIN-2013-20/</Prefix><KeyCount>1</KeyCount><MaxKeys>1000</MaxKeys><Delimiter>/</Delimiter><EncodingType>url</EncodingType><IsTruncated>false</IsTruncated><CommonPrefixes><Prefix>cc-index/table/cc-main/warc/crawl%3DCC-MAIN-2013-20/subset%3Dwarc/</Prefix></CommonPrefixes></ListBucketResult>'
2020-06-02 14:24:33,990 - MainThread - botocore.hooks - DEBUG - Event needs-retry.s3.ListObjectsV2: calling handler <bound method RetryHandler.needs_retry of <botocore.retries.standard.RetryHandler object at 0x7ff2e89b69e8>>
2020-06-02 14:24:33,990 - MainThread - botocore.retries.standard - DEBUG - Not retrying request.
2020-06-02 14:24:33,990 - MainThread - botocore.hooks - DEBUG - Event needs-retry.s3.ListObjectsV2: calling handler <bound method S3RegionRedirector.redirect_from_error of <botocore.utils.S3RegionRedirector object at 0x7ff2e89b66d8>>
2020-06-02 14:24:33,990 - MainThread - botocore.hooks - DEBUG - Event after-call.s3.ListObjectsV2: calling handler <function decode_list_object_v2 at 0x7ff2ea878268>
2020-06-02 14:24:33,991 - MainThread - botocore.hooks - DEBUG - Event after-call.s3.ListObjectsV2: calling handler <function enhance_error_msg at 0x7ff2e8f5a1e0>
2020-06-02 14:24:33,991 - MainThread - botocore.hooks - DEBUG - Event after-call.s3.ListObjectsV2: calling handler <bound method RetryQuotaChecker.release_retry_quota of <botocore.retries.standard.RetryQuotaChecker object at 0x7ff2e89b6630>>
PRE subset=warc/