Thanks to stackoverflow, i managed to copy hadoop-aws-2.7.3.jar and aws-java-sdk-1.7.4.jar from maven repo into $SPARK_HOME/jars/ to get s3a:// going for reading from S3 buckets using pyspark (spark 2.2.0) on my ec2 linux instance.
df=spark.read.option("header","true").csv("s3a://bucket/csv_file")
But I'm stuck at writing the transformed data back into s3 bucket with server side encryption enabled. As expected below action throws "Access Denied" as I haven't specified flag to enable server side encryption within pyspark execution env
df.write.parquet("s3a://s3_bucket/output.parquet")
To verify, I wrote to a local file and uploaded to s3 bucket using -sse and this works fine
aws s3 cp local_path s3://s3_bucket/ --sse
How do I enable server side encryption in pyspark similar to above?
Note: I did try adding "fs.s3a.enableServerSideEncryption true" to spark-default.conf and passing the same via --conf parameter of pyspark at start but no joy.
Thanks