AWS IAM Role for Service Account with EBS CSI driver: could not create volume in EC2: NoCredentialProviders: no valid providers in chain

user_19768823 0
Viewed 205

I have successfully installed the AWS EBS CSI driver to my EKS cluster.

This is meant to be using the "IAM Role for Service Account" technique.

I am trying to utilise the checkout example app that AWS have given here The pod will not come up (pending) and the PVC is showing this:

Name:          ebs-claim
Namespace:     test
StorageClass:  ebs-sc
Status:        Pending
Volume:        
Labels:        app=ebs-claim
               com.mylabel.contact=dl-myteam.dlonp1
Annotations:   volume.beta.kubernetes.io/storage-provisioner: ebs.csi.aws.com
               volume.kubernetes.io/selected-node: ip-10-232-100-115.ec2.internal
Finalizers:    [kubernetes.io/pvc-protection]
Capacity:      
Access Modes:  
VolumeMode:    Filesystem
Used By:       meme-ebs
Events:
  Type     Reason              Age               From                                                                                      Message
  ----     ------              ----              ----                                                                                      -------
  Warning  ProvisioningFailed  27s               persistentvolume-controller                                                               storageclass.storage.k8s.io "ebs-sc" not found
  Normal   Provisioning        8s (x4 over 25s)  ebs.csi.aws.com_ebs-csi-controller-6dfdb77cdf-fbsbz_1760973c-09bb-43ab-b005-ffcd818447fc  External provisioner is provisioning volume for claim "test/ebs-claim"
  Warning  ProvisioningFailed  5s (x4 over 22s)  ebs.csi.aws.com_ebs-csi-controller-6dfdb77cdf-fbsbz_1760973c-09bb-43ab-b005-ffcd818447fc  failed to provision volume with StorageClass "ebs-sc": rpc error: code = Internal desc = Could not create volume "pvc-05efbff8-9506-4003-9bab-e1ce4719bc1c": could not create volume in EC2: NoCredentialProviders: no valid providers in chain
caused by: EnvAccessKeyNotFound: failed to find credentials in the environment.
SharedCredsLoad: failed to load profile, .
EC2RoleRequestError: no EC2 instance role found
caused by: EC2MetadataError: failed to make EC2Metadata request

Similar to an issue I saw here, but had no answers.

Can anyone suggest things to try? Seems like the IAM role is not wired thru to the API that mounts the volume on EC2?

amazon-eks amazon-web-services amazon-ec2
2 Answers

Looks like an issue with the service account that your efs csi driver is using. For example, make sure it's using the right role with the right trust policy for your EKS cluster. For example check the right annotation below:

apiVersion: v1
kind: ServiceAccount
metadata:
  labels:
    app.kubernetes.io/name: aws-efs-csi-driver
  name: efs-csi-controller-sa
  namespace: kube-system
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::111122223333:role/AmazonEKS_EFS_CSI_DriverRole

And the role that you are using has the right trust policy

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Federated": "arn:aws:iam::111122223333:oidc-provider/oidc.eks.region-code.amazonaws.com/id/EXAMPLED539D4633E53DE1B71EXAMPLE"
      },
      "Action": "sts:AssumeRoleWithWebIdentity",
      "Condition": {
        "StringEquals": {
          "oidc.eks.region-code.amazonaws.com/id/EXAMPLED539D4633E53DE1B71EXAMPLE:sub": "system:serviceaccount:kube-system:efs-csi-controller-sa"
        }
      }
    }
  ]
}

The instructions here are pretty clear. (It worked for me)

user_29892610

Ensure that the created ServiceAccounts have the correct IRSA annotations.

If you are using the helm chart, and doing an upgrade from an older version, double check the location of the IRSA ServiceAccount annotation (they may have changed, had me stumped for a bit as to why things didn't work).

user_125020550
Related