Can use AWS CLI with credentials file but not with environment variables

user_7269570 0
Viewed 2487

I usually use my AWS CLI commands after setting a profile, with the environment variable AWS_PROFILE, with the ~/.aws/credentials file. This works.

What I'm currently trying to do is to set up access via environment variables. To do so, I'm setting those variables in my .bash_profile file - I literally copied the aws_access_key_id and aws_secret_access_key entries from the credentials files and put them in my bash_profile file, under the names of AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY.

The environment variables are being correctly set, and, yet, when I try to access AWS resources (in this case, I'm trying to run a ls S3 command over a bucket, so the region doesn't matter), I get the message

An error occurred (InvalidAccessKeyId) when calling the ListObjectsV2 operation: The AWS Access Key Id you provided does not exist in our records

which is very weird to me, since the keys are exactly the same. To confirm this, I switch to my credential profile with the AWS_PROFILE environment variable, and then the command works normally.

I suspected that, somehow, I was setting the wrong environment variables, or something like that. Then, I read this AWS guide, and ran the command aws configure list, which, in the first case (the case with environment variables only), returned

      Name                    Value             Type    Location
      ----                    -----             ----    --------
   profile                <not set>             None    None
access_key     ****************AAAA              env
secret_key     ****************AAAA              env
    region                us-east-1              env    ['AWS_REGION', 'AWS_DEFAULT_REGION']

For the second case (with the profile set), it returned

     Name                    Value             Type    Location
      ----                    -----             ----    --------
   profile              dev-staging           manual    --profile
access_key     ****************AAAA shared-credentials-file
secret_key     ****************AAAA shared-credentials-file
    region                us-east-1              env    ['AWS_REGION', 'AWS_DEFAULT_REGION']

In other words, the environment variables are being correctly set, the AWS CLI acknowledges them, their values are the same as when they are set via the credentials file, and, yet, for some reason, it doesn't work that way.

I thought it could be due to the aws_session_token, which I also tried to set as an environment variable, to no avail.

I need to access AWS resources this way to simulate the environment in which my code will run, and I don't see why this would not work the way I'm intending.

Any ideas on how to solve it are appreciated.

aws-cli amazon-web-services amazon-s3
2 Answers

You need to edit your ~/.aws/config file when you would like to refer to the credentials from environment variables instead of credentials file.

With AWS Access Keys in credentials file, you must be having profile setup as OR there is no such source_profile config for any profile:

[default]
source_profile = default

However, when you would like to use the credentials set in your environment variables or bash_profile, change/add this setting to every profile in your config file:

[default]
credential_source = Environment

With this change, it should work with your Environment variables as well.

In case you've multiple profiles in ~/.aws/config file, just replace/add source_profile = <profile-name> with credential_source = Environment

user_51118700

In case someone stumbles on this, a possible culprit for this might be the AWS_SESSION_TOKEN and AWS_SECURITY_TOKEN environment variables.

If you were using different AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY environment variables before and an AWS CLI command is run, directly or indirectly, then after first auth the above two token variables are set. And after we overwrite the existing AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY with new values, the older token variables still remain as they are and AWS CLI doesn't have an explicit check to see if the access/secret keys were updated and it continues to use older tokens resulting in older keys being used internally and will continue to do so till the tokens expire. The aws configure will continue to show new access keys, but internally it would be using older access keys because of cached tokens.

So if you want to continue to use the environment variables in such scenarios you will need to unset the two environment variables containing tokens and in your case also add an unset command for two token variables after setting the new access/secret keys in environment variables.

unset AWS_SESSION_TOKEN
unset AWS_SECURITY_TOKEN

This behavior is one of the reasons people prefer to use different profiles either using aws configure or editing the ~/.aws/* files, and explicitly specify them using the --profile in commands instead of using environment variables.

Per the AWS cli configuration precedence order the usage of ~/.aws/config file is at the top of the precedence order of where AWS CLI picks up the auth to be used, so it overrides the token environment variables and works in your case.

user_86770710
Related