We use spot instances in ECS from AWS Batch. Regularly while running inside the container we call to the AWS CLI to upload a local directory to S3 (aws s3 cp ...). We have a signal handler for SIGTERM that also calls this to upload the latest changes just before spot termination.
While the upload to S3 works fine during normal operation the one after SIGTERM always fails with:
Error when retrieving credentials from container-role: Error retrieving metadata: Received error when attempting to retrieve ECS metadata: Connect timeout on endpoint URL: "http://169.254.170.2/v2/credentials/*************"
Timestamps on logging show this timeout is reported about 10 seconds after SIGTERM (which is probably the timeout in the AWS CLI?).
Why is the ECS metadata service unavailable while the instance is draining? Is there a way around this?