How to write a spark dataframe to S3 with MD5 header?

Viewed 242

I have a Spark DataFrame that I need to write to an S3 Object Lock enabled bucket. A simple write results in this error

df.write.parquet(output_path)

com.amazonaws.services.s3.model.AmazonS3Exception: 
Content-MD5 HTTP header is required for Put Object requests with Object Lock parameters

Any ideas how can I solve this?

There are ways to work this with boto3 type uploads, but how to do it with spark.df.write

s3_client.put_object(
    Bucket=<S3_BUCKET_NAME>, 
    Key=<KEY>, 
    Body=open(<FILE_NAME>, "rb"),
    ContentMD5=<MD5_HASH>
)
1 Answers

not supported in hadoop's s3a connector I'm afraid.

As is usual in OSS projects, your participation will help there: https://issues.apache.org/jira/browse/HADOOP-15224. That JIRA was created in 2018 and hasn't had attention as nobody new it was actually needed. We could maybe revisit that.

Don't know about the EMR s3 connector.

Related