S3 connectivity issue when running Elasticsearch in Kubernetes with ServiceAccount to IAM role mapping

user_1035556 0
Viewed 849

I faced an issue when I tried to connect an Elasticsearch 7.8.0 instance running in Kubernetes to S3 bucket as backup storage. The installation is the following.

There is an Elasticsearch deployed as a custom resource (ECK). Then there is a trusted relationship between a ServiceAccount that is used to execute pods deployed by this resource and IAM role which has permissions to access the target bucket for backups. Then I created a snapshot repository of AWS S3 type and provided that bucket as a target. But when I tried to validate the repository, I received an Access Denied error.

As far as I see according to ECK original documentation Create automated snapshots it is assumed that the credentials to access the bucket are stored via Keystore. But I also see that the AWS SDK used by my version of Elasticsearch should support obtaining credentials from ServiceAccount: https://github.com/elastic/elasticsearch/blob/v7.8.0/plugins/repository-s3/build.gradle#L31 (Using a supported AWS SDK)

Therefore I am wondering why these components are not working as expected. Did anyone try to establish a similar configuration? Are there any fundamental issues in Elasticsearch that prevent it from obtaining credentials from the ServiceAccount?

I tried to describe the case in general since the number of entities involved is relatively large. Please, let me know if this case requires extra details about the entities' configuration.

--- UPD

Here is a sample of a pod spec:

kind: Pod
apiVersion: v1
metadata:
  name: logging-elasticsearch-es-default-0
  generateName: logging-elasticsearch-es-default-
  namespace: elastic-logging
  selfLink: /api/v1/namespaces/elastic-logging/pods/logging-elasticsearch-es-default-0
  uid: 29ff69cc-9c01-448d-821e-d0da26cd2ba6
  resourceVersion: '1950203'
  creationTimestamp: '2020-07-15T15:38:40Z'
  labels:
    common.k8s.elastic.co/type: elasticsearch
    controller-revision-hash: logging-elasticsearch-es-default-78fd9d9d45
    elasticsearch.k8s.elastic.co/cluster-name: logging-elasticsearch
    elasticsearch.k8s.elastic.co/config-hash: '2541120219'
    elasticsearch.k8s.elastic.co/http-scheme: https
    elasticsearch.k8s.elastic.co/node-data: 'true'
    elasticsearch.k8s.elastic.co/node-ingest: 'true'
    elasticsearch.k8s.elastic.co/node-master: 'true'
    elasticsearch.k8s.elastic.co/node-ml: 'true'
    elasticsearch.k8s.elastic.co/statefulset-name: logging-elasticsearch-es-default
    elasticsearch.k8s.elastic.co/version: 7.8.0
    statefulset.kubernetes.io/pod-name: logging-elasticsearch-es-default-0
  annotations:
    co.elastic.logs/module: elasticsearch
    kubernetes.io/psp: eks.privileged
    update.k8s.elastic.co/timestamp: '2020-07-15T15:42:46.115776158Z'
  ownerReferences:
    - apiVersion: apps/v1
      kind: StatefulSet
      name: logging-elasticsearch-es-default
      uid: 34a0109c-708b-402c-96b9-89c3e292a6a6
      controller: true
      blockOwnerDeletion: true
spec:
  volumes:
    - name: aws-iam-token
      projected:
        sources:
          - serviceAccountToken:
              audience: sts.amazonaws.com
              expirationSeconds: 86400
              path: token
        defaultMode: 420
  containers:
    - name: elasticsearch
      image: 'docker.elastic.co/elasticsearch/elasticsearch:7.8.0'
      env:
        - name: AWS_ROLE_ARN
          value: 'arn:aws:iam::<account_id>:role/AWSS3ElkSnapshotBucket'
        - name: AWS_WEB_IDENTITY_TOKEN_FILE
          value: /var/run/secrets/eks.amazonaws.com/serviceaccount/token
      volumeMounts:
        - name: aws-iam-token
          readOnly: true
          mountPath: /var/run/secrets/eks.amazonaws.com/serviceaccount

--- UPD 2

Here is a sample of ServiceAccount that is used to executed the pod.

kind: ServiceAccount
apiVersion: v1
metadata:
  name: default
  namespace: elastic-logging
  selfLink: /api/v1/namespaces/elastic-logging/serviceaccounts/default
  uid: 7e97ac47-d893-4425-9da7-ef87848714e8
  resourceVersion: '1894373'
  creationTimestamp: '2020-07-07T13:08:01Z'
  annotations:
    eks.amazonaws.com/role-arn: 'arn:aws:iam::<account_id>:role/AWSS3ElkSnapshotBucket'
secrets:
  - name: default-token-q5qft
amazon-eks elasticsearch kubernetes
1 Answers

So, looks like the official docs show how to tie it with a service account using GCS which makes sense since GCP also has this concept of service accounts.

For AWS is a little bit different you typically associate a role or assume a role that has the right permissions. This is a good guide on how to use it for Kubernetes and how it's officially supported now. I believe the ECK operator doesn't support it yet since the docs say that the S3 storage plugin either uses the EC2 or ECS IAM roles. This is odd because you'd want to use an S3 type of role to access S3.

What's in the docs is how to use an AWS user instead of a role to access S3:

If you configure your plugin that way, you should be able to access S3 provided that the user has write access to the S3 bucket where you want to store the snapshots.

user_29892610
Related