EMR Serverless Read From S3 Cross Account

Viewed 24

I have an EMR Application that runs inside a VPC in a Private Subnet with NAT on. This application can read files from the buckets in the same account, but when it tries to read a file in a bucket in other aws account it gets access denied.

The emr applications has the following policy:

S3FullAccess:
    {
        "Version": "2012-10-17",
        "Statement": [
            {
                "Effect": "Allow",
                "Action": [
                    "s3:*",
                    "s3-object-lambda:*"
                ],
                "Resource": "*"
            }
        ]
    }

EMR Spark packages settings:

--conf spark.jars.packages=org.apache.hadoop:hadoop-aws:3.2.0

Job Spark:

from pyspark.sql import SparkSession
spark = (SparkSession.builder
             .config("spark.hadoop.fs.s3a.fast.upload", True)
             .config("spark.hadoop.fs.s3a.impl", "org.apache.hadoop.fs.s3a.S3AFileSystem")
             .config("spark.sql.adaptive.enabled", "true")
             .config("spark.jars.packages", "org.apache.hadoop:hadoop-aws:3.2.0")
             .enableHiveSupport().getOrCreate()
             )

df_from_another_s3_account_bucket = spark.read.parquet('s3a://bucket_account_b/path/file.parquet')

This execution is returning the error:

 java.nio.file.AccessDeniedException: s3a://bucket_account_b/path/file.parquet: getFileStatus on  s3a://bucket_account_b/path/file.parquet: com.amazonaws.services.s3.model.AmazonS3Exception: Forbidden (Service: Amazon S3; Status Code: 403; Error Code: 403 Forbidden;

I have tried using credentials in the spark configs:

.config("fs.s3a.access.key", key_bucket_account_b)
.config("fs.s3a.secret.key", secret_bucket_account_b)

I have also tried creating a bucket policy allowing the emr aws account, but still got the same error.

{
    "Version": "2012-10-17",
    "Id": "Policy1543283",
    "Statement": [
        {
            "Sid": "Stmt1412820423",
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::emr_aws_account:root"
            },
            "Action": "s3:*",
            "Resource": "arn:aws:s3:::bucket_account_b"
        }
    ]
}

What else could I try?

0 Answers
Related