com.amazonaws.AmazonClientException: Unable to execute HTTP request: No such host is known (spark-tunes.s3a.ap-south-1.amazonaws.com)

user_12857113 0
Viewed 1345

I am trying to read a json file stored in S3 bucket from spark in local mode via pycharm. But I'm getting the below error message:

"py4j.protocol.Py4JJavaError: An error occurred while calling o37.json. : com.amazonaws.AmazonClientException: Unable to execute HTTP request: No such host is known (spark-tunes.s3a.ap-south-1.amazonaws.com)"

(spark-tunes is my S3 bucket name).

Below is the code I executed. Please help me to know if I'm missing something.

spark = SparkSession.builder.appName('DF Read').config('spark.master', 'local').getOrCreate()

spark._jsc.hadoopConfiguration().set("fs.s3a.access.key", "access_key")
spark._jsc.hadoopConfiguration().set("fs.s3a.secret.key", "secret_key")

spark._jsc.hadoopConfiguration().set("fs.s3a.endpoint", "s3a.ap-south-1.amazonaws.com")
spark._jsc.hadoopConfiguration().set("com.amazonaws.services.s3a.enableV4", "true")
spark._jsc.hadoopConfiguration().set("fs.s3a.impl", "org.apache.hadoop.fs.s3a.S3AFileSystem")

df = spark.read.json("s3a://bucket-name/folder_name/*.json")

df.show(5)
pyspark dataframe amazon-s3 python
1 Answers

try setting fs.s3a.path.style.access to false and instead of prefixing the bucket name to the host, the aws s3 client will use paths under the endpoint

also: drop the fs.s3a.impl line. That is superstition passed down across stack overflow examples. It's not needed. really.

user_22612740
Related