IntelliJ: setting AWS env vars for gradle run config

user_7619689 0
Viewed 2931

I'm trying to set up my local dev env for a Java Spring Boot configured with gradle which needs a working AWS CLI environment. It's important to note that there are different profiles (.aws/config) present (default not configured) which are different role switches (role_arn) with MFA (mfa_serial).

Description

At the moment I'm using Windows 10 Build 18363 ("19.09"), IntelliJ Ultimate 20.02, gradle 5.6.4. In order to run this application I've set up a run configuration with the gradle clean bootRun task and the following env vars:

  • AWS_SECRET_ACCESS_KEY
  • AWS_ACCESS_KEY_ID
  • AWS_REGION
  • AWS_DEFAULT_REGION
  • AWS_PROFILE
  • AWS_DEFAULT_PROFILE
#.aws/config
[profile prod]
region=eu-central-1
output=json
role_arn=<role_arn_prod>
source_profile=site-iam
mfa_serial=<arn_iam_user_mfa_1>


[profile sit]
region=eu-central-1
output=json
role_arn=<role_arn_sit>
source_profile=site-iam
mfa_serial=<arn_iam_user_mfa_1>

[profile dev]
region=eu-central-1
output=json
role_arn=<role_arn_dev>
source_profile=site-iam
mfa_serial=<arn_iam_user_mfa_1>

[profile site-iam]
region=eu-central-1
output=json

[default]


#.aws/credentials
[default]
aws_access_key_id = <access_key_id_1>
aws_secret_access_key = <secret_access_key_1>

[site-iam]
aws_access_key_id = <access_key_id_1>
aws_secret_access_key = <secret_access_key_1>

Behaviour

The request inside of this java application is quite simple.

GetObjectRequest getObjectRequest = GetObjectRequest.builder()
    .bucket(this.bucket)
    .key(this.key)
    .build();
String response = this
    .s3Client
    .getObjectAsBytes(getObjectRequest).asString(Charset.forName(StandardCharsets.UTF_8.name()));

The error below suggests that there was no role switch and / or the authentication failed (because we need to submit the TOTP in order to authenticate.

Caused by: software.amazon.awssdk.services.s3.model.S3Exception: Access Denied (Service: S3, Status Code: 403, Request ID: <REQUEST_ID>)
    at software.amazon.awssdk.core.internal.http.CombinedResponseHandler.handleErrorResponse(CombinedResponseHandler.java:123) ~[sdk-core-2.13.13.jar:na]
    at software.amazon.awssdk.core.internal.http.CombinedResponseHandler.handleResponse(CombinedResponseHandler.java:79) ~[sdk-core-2.13.13.jar:na]
    at software.amazon.awssdk.core.internal.http.CombinedResponseHandler.handle(CombinedResponseHandler.java:59) ~[sdk-core-2.13.13.jar:na]
    at software.amazon.awssdk.core.internal.http.CombinedResponseHandler.handle(CombinedResponseHandler.java:40) ~[sdk-core-2.13.13.jar:na]
    at software.amazon.awssdk.core.internal.http.pipeline.stages.HandleResponseStage.execute(HandleResponseStage.java:40) ~[sdk-core-2.13.13.jar:na]
    at software.amazon.awssdk.core.internal.http.pipeline.stages.HandleResponseStage.execute(HandleResponseStage.java:30) ~[sdk-core-2.13.13.jar:na]
    at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206) ~[sdk-core-2.13.13.jar:na]
    at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptTimeoutTrackingStage.execute(ApiCallAttemptTimeoutTrackingStage.java:73) ~[sdk-core-2.13.13.jar:na]
    at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptTimeoutTrackingStage.execute(ApiCallAttemptTimeoutTrackingStage.java:42) ~[sdk-core-2.13.13.jar:na]
    at software.amazon.awssdk.core.internal.http.pipeline.stages.TimeoutExceptionHandlingStage.execute(TimeoutExceptionHandlingStage.java:77) ~[sdk-core-2.13.13.jar:na]
    at software.amazon.awssdk.core.internal.http.pipeline.stages.TimeoutExceptionHandlingStage.execute(TimeoutExceptionHandlingStage.java:39) ~[sdk-core-2.13.13.jar:na]
    at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage.execute(RetryableStage.java:64) ~[sdk-core-2.13.13.jar:na]
    at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage.execute(RetryableStage.java:34) ~[sdk-core-2.13.13.jar:na]
    at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206) ~[sdk-core-2.13.13.jar:na]
    at software.amazon.awssdk.core.internal.http.StreamManagingStage.execute(StreamManagingStage.java:56) ~[sdk-core-2.13.13.jar:na]
    at software.amazon.awssdk.core.internal.http.StreamManagingStage.execute(StreamManagingStage.java:36) ~[sdk-core-2.13.13.jar:na]
    at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.executeWithTimer(ApiCallTimeoutTrackingStage.java:80) ~[sdk-core-2.13.13.jar:na]
    at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.execute(ApiCallTimeoutTrackingStage.java:60) ~[sdk-core-2.13.13.jar:na]
    at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.execute(ApiCallTimeoutTrackingStage.java:42) ~[sdk-core-2.13.13.jar:na]
    at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206) ~[sdk-core-2.13.13.jar:na]
    at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206) ~[sdk-core-2.13.13.jar:na]
    at software.amazon.awssdk.core.internal.http.pipeline.stages.ExecutionFailureExceptionReportingStage.execute(ExecutionFailureExceptionReportingStage.java:37) ~[sdk-core-2.13.13.jar:na]
    at software.amazon.awssdk.core.internal.http.pipeline.stages.ExecutionFailureExceptionReportingStage.execute(ExecutionFailureExceptionReportingStage.java:26) ~[sdk-core-2.13.13.jar:na]
    at software.amazon.awssdk.core.internal.http.AmazonSyncHttpClient$RequestExecutionBuilderImpl.execute(AmazonSyncHttpClient.java:189) ~[sdk-core-2.13.13.jar:na]
    at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.invoke(BaseSyncClientHandler.java:121) ~[sdk-core-2.13.13.jar:na]
    at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.doExecute(BaseSyncClientHandler.java:147) ~[sdk-core-2.13.13.jar:na]
    at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.execute(BaseSyncClientHandler.java:76) ~[sdk-core-2.13.13.jar:na]
    at software.amazon.awssdk.core.client.handler.SdkSyncClientHandler.execute(SdkSyncClientHandler.java:52) ~[sdk-core-2.13.13.jar:na]
    at software.amazon.awssdk.awscore.client.handler.AwsSyncClientHandler.execute(AwsSyncClientHandler.java:62) ~[aws-core-2.13.13.jar:na]
    at software.amazon.awssdk.services.s3.DefaultS3Client.getObject(DefaultS3Client.java:3606) ~[s3-2.13.13.jar:na]
    at software.amazon.awssdk.services.s3.S3Client.getObjectAsBytes(S3Client.java:7563) ~[s3-2.13.13.jar:na]

A similar behaviour is reproduciblevia the cmd

aws s3 ls

An error occurred (AccessDenied) when calling the ListBuckets operation: Access Denied

Different approaches

STS get-session-token | env var

I've obtained the session token via

aws sts get-session-token --serial <arn_iam_user_mfa_1> --token-code <TOTP> --duration-seconds 129600

as json

{
    "Credentials": {
        "AccessKeyId": "<access_key_id_2>",
        "SecretAccessKey": "<secret_access_key_2>",
        "SessionToken": "<session_token_2>",
        "Expiration": "2020-08-02T22:57:12+00:00"
    }
}

and set updated the following env vars to the obtained values:

  • AWS_ACCESS_KEY_ID: <access_key_id_2>
  • AWS_SECRET_ACCESS_KEY: <secret_access_key_2>
  • AWS_SESSION_TOKEN: <session_token_2>
  • AWS_SECURITY_TOKEN: <session_token_2>

but this is returning the same error

Caused by: software.amazon.awssdk.services.s3.model.S3Exception: Access Denied (Service: S3, Status Code: 403, Request ID: <request_id>)

Quick and dirty .aws/config fix

After several time of debugging the default profile was changed in the following way

[default]
region=eu-central-1
output=json
role_arn=<role_arn_prod>
source_profile=site-iam
mfa_serial=<arn_iam_user_mfa_1>

which resulted in

Caused by: software.amazon.awssdk.core.exception.SdkClientException: Unable to load credentials from any of the providers in the chain AwsCredentialsProviderChain(credentialsProviders=[SystemPropertyCredentialsProvider(), EnvironmentVariableCredentialsProvider(), WebIdentityTokenCredentialsProvider(), ProfileCredentialsProvider(), ContainerCredentialsProvider(), InstanceProfileCredentialsProvider()]) : [SystemPropertyCredentialsProvider(): Unable to load credentials from system settings. Access key must be specified either via environment variable (AWS_ACCESS_KEY_ID) or system property (aws.accessKeyId)., EnvironmentVariableCredentialsProvider(): Unable to load credentials from system settings. Access key must be specified either via environment variable (AWS_ACCESS_KEY_ID) or system property (aws.accessKeyId)., WebIdentityTokenCredentialsProvider(): Either the environment variable AWS_WEB_IDENTITY_TOKEN_FILE or the javaproperty aws.webIdentityTokenFile must be set., ProfileCredentialsProvider(): To use assumed roles in the '<ROLE_PROFILE>' profile, the 'sts' service module must be on the class path., ContainerCredentialsProvider(): Cannot fetch credentials from container - neither AWS_CONTAINER_CREDENTIALS_FULL_URI or AWS_CONTAINER_CREDENTIALS_RELATIVE_URI environment variables are set., InstanceProfileCredentialsProvider(): Unable to load credentials from service endpoint.]

Setting AWS_WEB_IDENTITY_TOKEN_FILE

The AWS_WEB_IDENTITY_TOKEN_FILE env var was set to an existing token file located in ~/.aws/cli/cache via a WSL sytle path (\wsl$\...) which leads to the same error as above even though the the env vars are present and / or updated

Caused by: software.amazon.awssdk.core.exception.SdkClientException: Unable to load credentials from any of the providers in the chain AwsCredentialsProviderChain(credentialsProviders=[SystemPropertyCredentialsProvider(), EnvironmentVariableCredentialsProvider(), WebIdentityTokenCredentialsProvider(), ProfileCredentialsProvider(), ContainerCredentialsProvider(), InstanceProfileCredentialsProvider()]) : [SystemPropertyCredentialsProvider(): Unable to load credentials from system settings. Access key must be specified either via environment variable (AWS_ACCESS_KEY_ID) or system property (aws.accessKeyId)., EnvironmentVariableCredentialsProvider(): Unable to load credentials from system settings. Access key must be specified either via environment variable (AWS_ACCESS_KEY_ID) or system property (aws.accessKeyId)., WebIdentityTokenCredentialsProvider(): Either the environment variable AWS_WEB_IDENTITY_TOKEN_FILE or the javaproperty aws.webIdentityTokenFile must be set., ProfileCredentialsProvider(): To use assumed roles in the '<ROLE_PROFILE>' profile, the 'sts' service module must be on the class path., ContainerCredentialsProvider(): Cannot fetch credentials from container - neither AWS_CONTAINER_CREDENTIALS_FULL_URI or AWS_CONTAINER_CREDENTIALS_RELATIVE_URI environment variables are set., InstanceProfileCredentialsProvider(): Unable to load credentials from service endpoint.]

Debugging

There are no permission problems. The download of this specific file works with the correct role arn.


C:\Users\tunnelblick>aws s3 ls

An error occurred (AccessDenied) when calling the ListBuckets operation: Access Denied

C:\Users\tunnelblick>aws s3 ls --profile prod
Enter MFA code for <arn_iam_user_mfa_1>:
2020-02-05 09:28:41 <bucket_1>
2020-06-23 05:16:07 <bucket_2>

C:\Users\tunnelblick>aws s3 cp s3://<bucket_1>/<file>.json .
fatal error: An error occurred (403) when calling the HeadObject operation: Forbidden

C:\Users\tunnelblick>aws s3 cp s3://<bucket_1>/<file>.json . --profile prod
download: s3://<bucket_1>/<file>.json to .\<file>.json

Setting these env vars in my WSL 1 terminal and executing aws s3 ls resulted in the same Access Denied error (An error occurred (AccessDenied) when calling the ListBuckets operation: Access Denied)

amazon-iam gradle amazon-web-services intellij-idea amazon-s3
1 Answers

Later on I also tried to set these values in the ~/.aws/credentials file. It seems that I was missing some options which could be solved with aws-mfa (a python wrapper for AWS STS). I was using the following example

aws-mfa --duration 1800 --device arn:aws:iam::123456788990:mfa/dudeman --assume-role arn:aws:iam::123456788990:role/some-role --role-session-name some-role-session`

This populates the default profile with additional options for a proper role switching like assumed_role, assume_role_arn, aws_session_token, aws_security_token, expiration and works like a charm with my run config.

user_76196890
Related