I have a Spark DataFrame that I need to write to an S3 Object Lock enabled bucket. A simple write results in this error
df.write.parquet(output_path)
com.amazonaws.services.s3.model.AmazonS3Exception:
Content-MD5 HTTP header is required for Put Object requests with Object Lock parameters
Any ideas how can I solve this?
There are ways to work this with boto3 type uploads, but how to do it with spark.df.write
s3_client.put_object(
Bucket=<S3_BUCKET_NAME>,
Key=<KEY>,
Body=open(<FILE_NAME>, "rb"),
ContentMD5=<MD5_HASH>
)