How to solve org.apache.spark.sql.AnalysisException: Path does not exist

Viewed 5418

I use Spark 2.2.0.

I use Spark to process datasets from S3. It worked fine until I decided to use wildcards in order to read data from sub-folders of the folder test.

val path = "s3://data/test"
val spark = SparkSession
  .builder()
  .appName("Test")
  .config("spark.sql.warehouse.dir", path)
  .enableHiveSupport()
  .getOrCreate()

import spark.implicits._    
val myData = spark.read.parquet(path + "/*/")

I get the following error:

17/11/20 18:54:21 ERROR ApplicationMaster: User class threw exception: org.apache.spark.sql.AnalysisException: Path does not exist: hdfs://ip-111-112-11-65.eu-west-1.compute.internal:8020/user/hdfs/s3/data/test/20171120/*;

I execute the above code with the following command:

spark-submit --deploy-mode cluster --driver-memory 10g

What I don't understand is why Spark tries to read from HDFS instead of reading from the provided path. The same piece of code works fine with another path like s3://data/test2/mytest.parquet.

0 Answers
Related