I use Spark 2.2.0.
I use Spark to process datasets from S3. It worked fine until I decided to use wildcards in order to read data from sub-folders of the folder test.
val path = "s3://data/test"
val spark = SparkSession
.builder()
.appName("Test")
.config("spark.sql.warehouse.dir", path)
.enableHiveSupport()
.getOrCreate()
import spark.implicits._
val myData = spark.read.parquet(path + "/*/")
I get the following error:
17/11/20 18:54:21 ERROR ApplicationMaster: User class threw exception: org.apache.spark.sql.AnalysisException: Path does not exist: hdfs://ip-111-112-11-65.eu-west-1.compute.internal:8020/user/hdfs/s3/data/test/20171120/*;
I execute the above code with the following command:
spark-submit --deploy-mode cluster --driver-memory 10g
What I don't understand is why Spark tries to read from HDFS instead of reading from the provided path. The same piece of code works fine with another path like s3://data/test2/mytest.parquet.