I use an Elastic Beanstalk environment for deploying a web application, and I've set up an IAM role for the instances the application will run on.
Everything works flawlessy 99.99% of the time, however intermittently I will see errors in our logs with request failures showing botocore errors like the following:
File "/opt/python/run/venv/local/lib/python3.6/site-packages/boto3/resources/factory.py", line 339, in property_loader
self.load()
File "/opt/python/run/venv/local/lib/python3.6/site-packages/boto3/resources/factory.py", line 505, in do_action
response = action(self, *args, **kwargs)
File "/opt/python/run/venv/local/lib/python3.6/site-packages/boto3/resources/action.py", line 83, in __call__
response = getattr(parent.meta.client, operation_name)(**params)
File "/opt/python/run/venv/local/lib/python3.6/site-packages/botocore/client.py", line 357, in _api_call
return self._make_api_call(operation_name, kwargs)
File "/opt/python/run/venv/local/lib/python3.6/site-packages/botocore/client.py", line 648, in _make_api_call
operation_model, request_dict, request_context)
File "/opt/python/run/venv/local/lib/python3.6/site-packages/botocore/client.py", line 667, in _make_request
return self._endpoint.make_request(operation_model, request_dict)
File "/opt/python/run/venv/local/lib/python3.6/site-packages/botocore/endpoint.py", line 102, in make_request
return self._send_request(request_dict, operation_model)
File "/opt/python/run/venv/local/lib/python3.6/site-packages/botocore/endpoint.py", line 132, in _send_request
request = self.create_request(request_dict, operation_model)
File "/opt/python/run/venv/local/lib/python3.6/site-packages/botocore/endpoint.py", line 116, in create_request
operation_name=operation_model.name)
File "/opt/python/run/venv/local/lib/python3.6/site-packages/botocore/hooks.py", line 356, in emit
return self._emitter.emit(aliased_event_name, **kwargs)
File "/opt/python/run/venv/local/lib/python3.6/site-packages/botocore/hooks.py", line 228, in emit
return self._emit(event_name, kwargs)
File "/opt/python/run/venv/local/lib/python3.6/site-packages/botocore/hooks.py", line 211, in _emit
response = handler(**kwargs)
File "/opt/python/run/venv/local/lib/python3.6/site-packages/botocore/signers.py", line 90, in handler
return self.sign(operation_name, request)
File "/opt/python/run/venv/local/lib/python3.6/site-packages/botocore/signers.py", line 157, in sign
auth.add_auth(request)
File "/opt/python/run/venv/local/lib/python3.6/site-packages/botocore/auth.py", line 357, in add_auth
raise NoCredentialsError
tocore.exceptions.NoCredentialsError: Unable to locate credentials
I don't know if this is just a race condition where IAM credentials are getting rotated (or something like that) and the instance's boto clients are trying to use them.
Any tips on remediating this or handling this failure gracefully?
P.S.
boto3==1.9.75
botocore==1.12.239
python==3.6.8