AWS CLI s3 copy fails with 403 error, trying to administrate a user-uploaded object

user_1335245 0
Viewed 32176

Trying to copy a file from an S3 bucket to my local machine:

aws s3 cp s3://my-bucket-name/audio-0b7ea3d0-13ab-4c7c-ac66-1bec2e572c14.wav ./

fatal error: An error occurred (403) when calling the HeadObject operation: Forbidden

Things I have confirmed:

  • I'm using version aws-cli/1.11.13 Python/3.5.2 Linux/4.4.0-75-generic botocore/1.4.70
  • The S3 Object key is correct. I have copied it directly from the S3 web interface.
  • The AWS CLI is configured with valid credentials. I generated a new key/secret pair. I deleted the ~/.aws folder before re-configuring the aws cli. The IAM web interface online confirms that the user specific by arn is in fact making use of S3 via the CLI.
  • The IAM user is granted the S3 full access managed policy, per this SO post. I removed all this users' policies, and then added only the AWS managed policy called AdministratorAccess, which includes "S3, Full access, All resources." Is there a different way to grant access via the CLI? I did not believe so.

Bucket policy is intended to grant wide open access:

    {
        "Sid": "AdminAccess",
        "Effect": "Allow",
        "Principal": "*",
        "Action": [
            "s3:*"
        ],
        "Resource": [
            "arn:aws:s3:::my-bucket-name",
            "arn:aws:s3:::my-bucket-name/*"
        ]
    }

How did I upload this object?

I uploaded this object using AWS Signature v4 signed upload policy from a web app in the client browser directly to AWS.

aws-cli amazon-web-services amazon-s3
5 Answers

In my case I have 3 accounts (A1, A2, A3) with 3 canonical users (canonical_user_account_A1, canonical_user_account_A2, canonical_user_account_A3) and 1 IAM role (R1) that is in A3.

Files are in a bucket in A2 and the files owner is canonical_user_account_A1 (this is on purpose). When I tried to list the files I didn't got any error, BUT when I tried to download one of them I got 

fatal error: An error occurred (403) when calling the HeadObject operation: Forbidden

I have added List and Get permissions for a R1 in the bucket policy and in the role permissions, in this case this is not enough, if the account were the bucket is not the owner it can't allow users from other account to get (download) files. So I needed to make sure that when I upload files I'm using:

    access_control_policy = {
    'Grants': [
        {
            'Grantee': {
                'ID': canonical_user_account_A2,
                'Type': 'CanonicalUser'
            },
            'Permission': 'READ'
        },
        {
            'Grantee': {
                'ID': canonical_user_account_A3,
                'Type': 'CanonicalUser'
            },
            'Permission': 'READ'
        },
    ],
    'Owner': {
        'ID': canonical_user_account_A1
    }
}

upload_extra_args = {'ACL': 'bucket-owner-full-control'}

s3_client.upload_file(file_path, bucket_name, s3_file_path, ExtraArgs=upload_extra_args)

s3_client.put_object_acl(AccessControlPolicy=access_control_policy, Bucket=bucket_name, Key=s3_file_path)

This allow both canonical_user_account_A2 and canonical_user_account_A3 to read and download the file.

user_5776520

In my case above error appeared when machine that was trying to contact S3 had system time far from the current one. Setting a correct time helped.

user_18446170
Related