I've spent quite a few hours now trying to get S3FS working with Pyspark 3.2 / Hadoop 3.3.1. I've gone through all sorts of ClassDefNotFound errors with the various compatibility issues that the hadoop-aws and aws-java-sdk jars have, but have now reached the point where I no longer get any exceptions when trying to read a file from S3 - instead the application just hangs indefinitely. This is my current configuration:
os.environ.update({'AWS_ACCESS_KEY_ID': 'SOME_SECRET_KEY',
'AWS_SECRET_ACCESS_KEY': 'SOME_SECRET_ACCESS_KEY'}
conf = SparkConf() \
.set('spark.jars.packages', 'org.apache.hadoop:hadoop-aws:3.3.1,com.amazonaws:aws-java-sdk-bundle:1.11.901') # also tried aws-java-sdk-bundle:1.12.96
sc = SparkContext(conf=conf)
spark = SparkSession(sc).builder.appName('test').getOrCreate()
spark._jsc.hadoopConfiguration().set("fs.s3a.aws.credentials.provider", "com.amazonaws.auth.EnvironmentVariableCredentialsProvider")
spark._jsc.hadoopConfiguration().set("fs.s3a.impl","org.apache.hadoop.fs.s3a.S3AFileSystem")
spark._jsc.hadoopConfiguration().set("fs.s3a.endpoint", "us-east-1.amazonaws.com")
print(f'pyspark hadoop version: {spark.sparkContext._jvm.org.apache.hadoop.util.VersionInfo.getVersion()}')
# executes this and a warning about missing hadoop-metrics2-s3a-file-system.properties, then hangs for good
df = spark.read.format('csv').load('s3a://bucket/file.csv').toPandas()
I've also tried the org.apache.spark:spark-hadoop-cloud_2.13:3.2.0 package instead of hadoop-aws, as suggested in the official spark docs here, but this also hangs in the same way. Prior to this every configuration I had tried usually resulted in java.lang.ClassNotFoundException: com.amazonaws.auth.AWSCredentialsProvider, so now I'm not sure if I'm getting hotter or colder.
Does anyone have a configuration they've used for Pyspark 3.2 / Hadoop 3.3.1 with S3FS access? Do I need to downgrade? I'm not that tied to any particular versions, although spark 3.0 + would be nice. Thanks in advance.