I am able to read/write files from spark standalone cluster to s3 using the below configuration.
val spark = SparkSession.builder()
.appName("Sample App")
.config("spark.master", "spark://spark:7077")
.config("spark.hadoop.fs.s3a.path.style.access", value = true)
.config("fs.s3a.fast.upload", value = true)
.config("fs.s3a.connection.ssl.enabled", value = false)
.config("mapreduce.fileoutputcommitter.algorithm.version", value = 2)
.config("spark.hadoop.fs.s3a.access.key", "Access Key Value")
.config("spark.hadoop.fs.s3a.secret.key", "Secret Key Value")
.config("spark.hadoop.fs.s3a.endpoint", "End-Point Value")
.getOrCreate()
But my requirement is to reuse the connection to s3 instead of mentioning s3 keys every time I create a spark-session. Like a mount point in data bricks.