Why only a root user can upload to S3 bucket running a Java program from a EC2 instance?

user_417291 0
Viewed 498

We have the problem in our application in uploading data to the S3 bucket crystal-dyn. This uploading works fine with the same program on the Centos-6 instances but fails on RHEL7 instances.

We have EC2 instances with attached role crystal-role. In turn this role has inline policy crystal-policy:

{
"RoleName": "crystal-role", 
"PolicyDocument": {
    "Version": "2012-10-17", 
    "Statement": [
        {
            "Action": [
                "ec2:DescribeInstances"
            ], 
            "Resource": "*", 
            "Effect": "Allow", 
            "Sid": "AllowDescribeInstances"
        }, 
        {
            "Action": [
                "s3:ListBucket", 
                "s3:GetBucketLocation"
            ], 
            "Resource": [
                "arn:aws:s3:::crystal-dyn" 
            ], 
            "Effect": "Allow", 
            "Sid": "AllowSeeLogBucket"
        }, 
        {
            "Action": [
                "s3:PutObject"
            ], 
            "Resource": [
                "arn:aws:s3:::crystal-dyn/*" 
            ], 
            "Effect": "Allow", 
            "Sid": "AllowPutLogs"
        }, 
        {
            "Action": [
                "kms:Encrypt", 
                "kms:GenerateDataKey"
            ], 
            "Resource": [
                "arn:aws:kms:us-east-1:566:key/a15912a107bb", 
                "arn:aws:kms:us-east-1:566201213358:key/158d81e9467a"
            ], 
            "Effect": "Allow", 
            "Sid": "AllowEncrypt"
        }, 
        {
            "Action": "sts:AssumeRole", 
            "Resource": "arn:aws:iam::389203956472:role/allow-cross-account-exec-api-qa2", 
            "Effect": "Allow", 
            "Sid": "AllowApiAccess"
        }
    ]
}, 
"PolicyName": "crystal-policy"
}

This policy should allow Java applications running on instances to upload data to the S3 bucket. However, this does not happen. So, we created a simple Java program to test uploading. Please, note that I use 3 versions of creating an AmazonS3 client. I run this from the command line:

java -cp ".:lib/*" org.examples.UploadObject

In the lib I have the following jars: 

aws-java-sdk-1.10.10.jar        aws-java-sdk-s3-1.11.339.jar  httpclient-4.5.5.jar  jackson-annotations-2.9.5.jar  jackson-databind-2.9.5.jar
aws-java-sdk-core-1.11.423.jar  commons-logging-1.1.3.jar     httpcore-4.4.9.jar    jackson-core-2.9.5.jar         joda-time-2.9.9.jar

The Java code:

   String clientRegion = "us-east-1";
    String bucketName = "crystal-dyn";
    String stringObjKeyName = "stringToUploadTest";
    try {
        AmazonS3 s3Client = AmazonS3ClientBuilder.standard().withCredentials(DefaultAWSCredentialsProviderChain.getInstance()).withRegion(clientRegion).build();
        System.out.println("s3Client=" + s3Client);
        s3Client.putObject(bucketName, stringObjKeyName, "Uploaded String Object");
        System.out.println("Uploading String  is done");
    }
    catch(AmazonServiceException e) {
        e.printStackTrace();
    }
    catch(SdkClientException e) {
        e.printStackTrace();
    }
try {
        System.out.println("Uploading to S3 bucket=" + bucketName + " string=" + stringObjKeyName  + " Building with No Creds");
        AmazonS3 s3Client = AmazonS3ClientBuilder.standard().withRegion(clientRegion).build();
        System.out.println("s3Client=" + s3Client);
        s3Client.putObject(bucketName, stringObjKeyName, "Uploaded String Object");
        System.out.println("Uploading String  is done");
    }
    catch(AmazonServiceException e) {
        e.printStackTrace();
    }
    catch(SdkClientException e) {
        e.printStackTrace();
    }
    try {
        System.out.println("Uploading to S3 bucket=" + bucketName + " string=" + stringObjKeyName  + " Building with No Creds and No region");
        AmazonS3 s3Client = AmazonS3ClientBuilder.standard().build();
        System.out.println("s3Client=" + s3Client);
        s3Client.putObject(bucketName, stringObjKeyName, "Uploaded String Object");
        System.out.println("Uploading String  is done");
    }
    catch(AmazonServiceException e) {
        e.printStackTrace();
    }
    catch(SdkClientException e) {
        e.printStackTrace();
    }

This programs works successfully only if it runs from the root user on the EC2 Linux instance, for all 3 versions of creating the AmazonS3 client. For all other users we got the output with the exception stack, see below. Please, note that no user including rood has the .aws directory with credentials. 

s3Client=com.amazonaws.services.s3.AmazonS3Client@682b2fa
com.amazonaws.SdkClientException: Unable to load AWS credentials from 
any provider in the chain: [EnvironmentVariableCredentialsProvider: 
Unable to load AWS credentials from environment variables (AWS_ACCESS_KEY_ID (or AWS_ACCESS_KEY) and AWS_SECRET_KEY (or AWS_SECRET_ACCESS_KEY)), SystemPropertiesCredentialsProvider: Unable to load AWS credentials from Java system properties (aws.accessKeyId and aws.secretKey), com.amazonaws.auth.profile.ProfileCredentialsProvider@20d525: profile file cannot be null, com.amazonaws.auth.EC2ContainerCredentialsProviderWrapper@3f56875e: Unable to load credentials from service endpoint]
at com.amazonaws.auth.AWSCredentialsProviderChain.getCredentials(AWSCredentialsProviderChain.java:136)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.getCredentialsFromContext(AmazonHttpClient.java:1186)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.runBeforeRequestHandlers(AmazonHttpClient.java:776)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:726)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:719)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:701) 
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:669)
at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:651)
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:515)
at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4365)
at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4312)
at com.amazonaws.services.s3.AmazonS3Client.putObject(AmazonS3Client.java:1755)
at com.amazonaws.services.s3.AmazonS3Client.putObject(AmazonS3Client.java:3448)
at org.examples.UploadObject.main(UploadObject.java:20)
Uploading to S3 bucket=dynarch-ac-logs-malachite-dyn 
string=stringToUploadTest Building with No Creds
s3Client=com.amazonaws.services.s3.AmazonS3Client@740773a3
com.amazonaws.services.s3.model.AmazonS3Exception: Access Denied (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied; Request ID: AF2735F42CCB60D0; S3 Extended Request ID: 7VyScO6XOs00oB/g0k8bqG3X3Ib01n4uT1xg8/2U72TCOKg8YKNIVgQrjjnF6XzUAfoB24wcYZY=), S3 Extended Request ID: 7VyScO6XOs00oB/g0k8bqG3X3Ib01n4uT1xg8/2U72TCOKg8YKNIVgQrjjnF6XzUAfoB24wcYZY=
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1660)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1324)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1074)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:745)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:719)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:701)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:669)
at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:651)
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:515)
at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4365)
at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4312)
at com.amazonaws.services.s3.AmazonS3Client.putObject(AmazonS3Client.java:1755)
at com.amazonaws.services.s3.AmazonS3Client.putObject(AmazonS3Client.java:3448)
at org.examples.UploadObject.main(UploadObject.java:33)
Uploading to S3 bucket=dynarch-ac-logs-malachite-dyn 
string=stringToUploadTest Building with No Creds and No region
com.amazonaws.SdkClientException: Unable to find a region via the region provider chain. Must provide an explicit region in the builder or setup environment to supply a region.
at com.amazonaws.client.builder.AwsClientBuilder.setRegion(AwsClientBuilder.java:436)
at com.amazonaws.client.builder.AwsClientBuilder.configureMutableProperties(AwsClientBuilder.java:402)
at com.amazonaws.client.builder.AwsSyncClientBuilder.build(AwsSyncClientBuilder.java:46)
at org.examples.UploadObject.main(UploadObject.java:44)
amazon-web-services amazon-ec2 amazon-s3
2 Answers

I am just thinking, instead of removing the rule. Can you try following if it resolves the issue?.(I would like to test it myself if I have your sample project). Usually, I use below steps to setup containers which interacts with S3 without having root permission and use IAM roles.

https://docs.aws.amazon.com/AmazonECS/latest/developerguide/ecs-agent-install.html

sudo sh -c "echo 'net.ipv4.conf.all.route_localnet = 1' >> /etc/sysctl.conf"
sudo sysctl -p /etc/sysctl.conf

sudo iptables -t nat -A PREROUTING -p tcp -d 169.254.170.2 --dport 80 -j DNAT --to-destination 127.0.0.1:51679
sudo iptables -t nat -A OUTPUT -d 169.254.170.2 -p tcp -m tcp --dport 80 -j REDIRECT --to-ports 51679

sudo sh -c 'iptables-save > /etc/iptables/rules.v4'
//OR
sudo sh -c 'iptables-save > /etc/sysconfig/iptables'
user_50307090

It appeared that the problem is in the iptables rule that allows the access only to the root.

iptables -L | grep root
DROP       all  --  anywhere             instance-data.ec2.internal  ! owner UID match root

Once this rule is removed uploading is working for all users

user_4172910
Related