Cannot write to EFS mounted on container on EC2 and ECS

user_2440505 0
Viewed 197

I have looked for many posts and answers with similar problems, but I haven't been able to find a solution to my problem. Tried many things, and even AWS's support isn't helping much.

Below are some excerpts from CFN template that builds the ECS cluster where the EFS should be mounted and used by the containers. It mounts, but even with root, I can't write to it.

  EFSVolume:
    DeletionPolicy: Retain
    UpdateReplacePolicy: Retain
    Type: AWS::EFS::FileSystem
    Properties:
      PerformanceMode: generalPurpose
      BackupPolicy: 
        Status: ENABLED
      Encrypted: true
      FileSystemPolicy:
        Version: "2012-10-17"
        Statement:
          - Effect: Allow
            Principal:
              AWS: "*"
            Action:
              - elasticfilesystem:ClientRootAccess
              - elasticfilesystem:ClientWrite
              - elasticfilesystem:ClientMount
            Condition:
              Bool:
                elasticfilesystem:AccessedViaMountTarget: 'true'
          - Effect: Deny
            Principal:
              AWS: "*"
            Action: "*"
            Condition:
              Bool:
                aws:SecureTransport: 'false'

...

  EFSAccessPoint:
    Type: AWS::EFS::AccessPoint
    Properties:
      FileSystemId: !Ref EFSVolume
      PosixUser:
        Uid: "1001"
        Gid: "1001"
      RootDirectory:
        CreationInfo:
          OwnerGid: "1001"
          OwnerUid: "1001"
          Permissions: "0777" #TODO review
        Path: "/"

  EFSTaskRole:
    Type: AWS::IAM::Role
    DeletionPolicy: Retain
    UpdateReplacePolicy: Retain
    Properties:
      AssumeRolePolicyDocument:
        Statement:
          - Action: sts:AssumeRole
            Effect: Allow
            Principal:
              Service: ecs-tasks.amazonaws.com
      Description: "Give access to ECS tasks to access the EFS Volume"
      ManagedPolicyArns:
        - arn:aws:iam::aws:policy/service-role/AmazonECSTaskExecutionRolePolicy
      Policies:
        - PolicyName: EFSAccessPoint
          PolicyDocument:
            Version: '2012-10-17'
            Statement:
              - Effect: Allow
                Action:
                  - elasticfilesystem:ClientMount
                  - elasticfilesystem:ClientWrite
                  - elasticfilesystem:ClientRootAccess
                Resource: !GetAtt EFSVolume.Arn
                Condition: #TODO ?? #CHECK
                  StringEquals:
                    elasticfilesystem:AccessPointArn: !GetAtt EFSAccessPoint.Arn

On the TaskDefinition:

  MountPoints:
    - ContainerPath: /data-amundsen
      SourceVolume: data-amundsen
      ReadOnly: false

...

  Volumes:
    - EFSVolumeConfiguration:
        AuthorizationConfig:
          AccessPointId: !Ref EFSAccessPoint
          IAM: ENABLED #CHECK
        FilesystemId: !Ref EFSVolume
        TransitEncryption: ENABLED
      Name: data-amundsen

And below is the result of my tests. Inside the container, as root, still I cannot write a single file to the volume.

[root@ab55f6d82abf /]# id
uid=0(root) gid=0(root) groups=0(root)
[root@ab55f6d82abf /]# df -h
Filesystem      Size  Used Avail Use% Mounted on
overlay          30G  4.7G   26G  16% /
tmpfs            64M     0   64M   0% /dev
tmpfs           3.9G     0  3.9G   0% /sys/fs/cgroup
127.0.0.1:/     8.0E     0  8.0E   0% /data-amundsen
/dev/xvda1       30G  4.7G   26G  16% /etc/hosts
shm              64M     0   64M   0% /dev/shm
tmpfs           3.9G     0  3.9G   0% /proc/acpi
tmpfs           3.9G     0  3.9G   0% /proc/scsi
tmpfs           3.9G     0  3.9G   0% /sys/firmware
[root@ab55f6d82abf /]# ls -la
total 16
drwxr-xr-x   1 root root    49 Jun 24 12:36 .
drwxr-xr-x   1 root root    49 Jun 24 12:36 ..
-rwxr-xr-x   1 root root     0 Jun 24 12:36 .dockerenv
-rw-r--r--   1 root root 12082 Mar  5  2019 anaconda-post.log
lrwxrwxrwx   1 root root     7 Mar  5  2019 bin -> usr/bin
drwxr-xr-x   2 root root  6144 Jun 24 12:05 data-amundsen
drwxr-xr-x   5 root root   340 Jun 24 12:36 dev
drwxr-xr-x   1 root root    66 Jun 24 12:36 etc
drwxr-xr-x   2 root root     6 Apr 11  2018 home
lrwxrwxrwx   1 root root     7 Mar  5  2019 lib -> usr/lib
lrwxrwxrwx   1 root root     9 Mar  5  2019 lib64 -> usr/lib64
drwxr-xr-x   2 root root     6 Apr 11  2018 media
drwxr-xr-x   2 root root     6 Apr 11  2018 mnt
drwxr-xr-x   1 root root    20 Mar 21  2019 opt
dr-xr-xr-x 150 root root     0 Jun 24 12:36 proc
dr-xr-x---   2 root root   114 Mar  5  2019 root
drwxr-xr-x   1 root root    19 Mar 21  2019 run
lrwxrwxrwx   1 root root     8 Mar  5  2019 sbin -> usr/sbin
drwxr-xr-x   2 root root     6 Apr 11  2018 srv
dr-xr-xr-x  13 root root     0 Jun 24 12:22 sys
drwxrwxrwt   1 root root    79 Jun 24 12:36 tmp
drwxr-xr-x   1 root root    19 Mar  5  2019 usr
drwxr-xr-x   1 root root    30 Mar  5  2019 var
[root@ab55f6d82abf /]# cd /data-amundsen
[root@ab55f6d82abf data-amundsen]# echo "a" > a.txt
bash: a.txt: Permission denied

Any ideas?

amazon-efs amazon-ecs amazon-web-services amazon-ec2
1 Answers

It seems that the root directory creation settings are not being applied, because you are specifying 0777 as the permissions, but they are 0755 in your ls command output.

After reading the documentation for this setting, I believe you are using this incorrectly. Specifically this note in the documentation:

If the RootDirectory > Path specified does not exist, EFS creates the root directory using the CreationInfo settings when a client connects to an access point.

This indicates that the CreationInfo settings are only applied if the location specified does not already exist on the EFS volume. Since the / location is always going to exist on an EFS volume, I don't think your CreationInfo settings are ever going to be applied.

The possible solutions I see are:

  1. Change the Path: "/" setting to something like Path: "/my-data". So inside the EFS volume that folder will be created automatically, with the permissions you specified. It will be mounted as the root for that EFS Access Point, so you won't actually see a /my-data folder in your ECS app.

  2. Or perform a one-time fix by going back into the shell and running sudo chmod 777 /data-amundsen

user_130700
Related