Unable to load AWS credentials from any provider in the chain: WebIdentityTokenCredentialsProvider: Unable to locate specified web identity token file

user_5890004 0
Viewed 8880

I am deploying my service using aws eks fargate deployment.yaml file and my service will connnect to dynamodb and while loading the id it throws the error. Even though I have web token file by following command.

kubectl exec -n first-namespace first-deployment-fhghgj567kkk-257xq env | grep AWS

AWS_REGION=us-east-1
AWS_ROLE_ARN=arn:aws:iam::263011912432:role/firstRole
AWS_WEB_IDENTITY_TOKEN_FILE=/var/run/secrets/eks.amazonaws.com/serviceaccount/token

But still it throws the following error.

Caused by: com.amazonaws.SdkClientException: Unable to load AWS credentials from any provider in the chain: 
[EnvironmentVariableCredentialsProvider: 
Unable to load AWS credentials from environment variables (AWS_ACCESS_KEY_ID (or AWS_ACCESS_KEY) 
and AWS_SECRET_KEY (or AWS_SECRET_ACCESS_KEY)), 
SystemPropertiesCredentialsProvider: Unable to load AWS credentials from Java system properties (aws.accessKeyId and aws.secretKey), 
WebIdentityTokenCredentialsProvider: Unable to locate specified web identity token file: 
/var/run/secrets/eks.amazonaws.com/serviceaccount/token, 
com.amazonaws.auth.profile.ProfileCredentialsProvider@7b58e085: profile file cannot be null, 
com.amazonaws.auth.EC2ContainerCredentialsProviderWrapper@39cb0014: Failed to connect to service endpoint: ]

I ssh into container and checked the token appears there its still present.

Can anyone please help? Thanks

amazon-eks amazon-dynamodb aws-fargate amazon-web-services
3 Answers

I had a similar issue, in my case I had to upgrade the java sdk and add a service account, I have not tried to call dynamo myself, but s3 and lambdas are working fine. Here are the steps I followed, this is the docs link: https://docs.aws.amazon.com/eks/latest/userguide/enable-iam-roles-for-service-accounts.html

First, enable OIDC

eksctl utils associate-iam-oidc-provider --cluster cluster_name --approve

Then create and apply your service account, note the role must have access to any service you want to use (dynamo):

apiVersion: v1
kind: ServiceAccount
metadata:
  name: fargateserviceaccount
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::ACCOUNT:role/MY_ROLE_NAME

On your pods, define the service account and env variables:

spec:
  serviceAccountName: <ServiceAccountName> (kubectl get sa)
  containers:
    - name: java-api
      image: <AccountID>.dkr.ecr.us-east-1.amazonaws.com/<Repo>:<tag>
      env:
        - name: AWS_REGION
          value: us-east-1

Finally use WebIdentityTokenCredentialsProvider in your code

// Create Credentials provider using ENV variables -> Service account is used to run  Fargate
WebIdentityTokenCredentialsProvider awsCredentialProvider = WebIdentityTokenCredentialsProvider.builder()
        .roleArn(System.getenv("AWS_ROLE_ARN"))
        .roleSessionName(System.getenv("AWS_ROLE_SESSION_NAME"))
        .webIdentityTokenFile(System.getenv("AWS_WEB_IDENTITY_TOKEN_FILE"))
        .build();
// Build client using explicit credentials provider
AWSLambdaClientBuilder builder = AWSLambdaClientBuilder.standard()
        .withRegion(REGION)
        .withCredentials(awsCredentialProvider);
client = builder.build();
user_110912900

I had a similar issue when using Kubernetes and updating the SDK version fixed it.

The containers in your pods must use an AWS SDK version that supports assuming an IAM role via an OIDC web identity token file. AWS SDKs that are included in Linux distribution package managers may not be new enough to support this feature.

You can find the minimum SDK version supported here.

user_50674170

I faced the same issue but observed a logger

WebIdentityTokenCredentialsProvider(): To use web identity tokens, the 'sts' service module must be on the class path

Adding the dependency software.amazon.awssdk:sts solved my issue. Hence please try by adding the sts dependency of the right aws sdk version if above mentioned logger is present.

user_59305000
Related