How to re-use the s3 connection in spark standalone cluster?

Viewed 13

I am able to read/write files from spark standalone cluster to s3 using the below configuration.

val spark = SparkSession.builder()
    .appName("Sample App")
    .config("spark.master", "spark://spark:7077")
    .config("spark.hadoop.fs.s3a.path.style.access", value = true)
    .config("fs.s3a.fast.upload", value = true)
    .config("fs.s3a.connection.ssl.enabled", value = false)
    .config("mapreduce.fileoutputcommitter.algorithm.version", value = 2)
    .config("spark.hadoop.fs.s3a.access.key", "Access Key Value")
    .config("spark.hadoop.fs.s3a.secret.key", "Secret Key Value")
    .config("spark.hadoop.fs.s3a.endpoint", "End-Point Value")
    .getOrCreate()

But my requirement is to reuse the connection to s3 instead of mentioning s3 keys every time I create a spark-session. Like a mount point in data bricks.

0 Answers
Related